This afternoon, Peter Kendall will take delivery of a new tractor. An arable farmer in Bedfordshire, who also manages to shoehorn his duties as president of the National Farmers’ Union into what’s already a busy week, he has had a good year. Just how good can be gauged by the cost of the tractor: £110, 000 for what Kendall describes as a “decent bit of kit”, equipped with hands-free, satellite-guided steering to ensure that furrows are precise to within a few centimetres of overlap. The two combines in the machine shed – caterpillar tracks at the front, giant wheels at the back – are worth even more. You won’t get one of these babies under £250,000 – if you can get one at all.

Because it isn’t only in Bedfordshire that farming has done well. It is booming in China, Brazil, Poland, western Europe and the US, not to mention the other cereal-growing counties of the UK. Order a specialist, top-of-the range John Deere tractor today, and it won’t arrive on your farm until 2010. No wonder, when Kendall passes his wife in a lane, they lean out of their respective car windows to discuss carpets. “Typical rich farmer building a house extension,” laughs Kendall, adding quickly: “Only a small extension. The children have run out of room.”

You won’t often get farmers to admit they are making money, particularly at present. You can understand why. Just as house prices are falling, some of them are spending the equivalent of a house on buying a single piece of machinery. Who pays for it? Ultimately, the shopper at the supermarket checkout, whose food bills reflect the dizzy price of wheat and the taxpayer, who is still paying the likes of Kendall £70 an acre to keep his land in production (in the bad times he might have given up without it). It is a sensitive time to be profitable, but there’s no getting away from it. As Chris Evans of the Agricultural Engineers Association, the trade body for farm-machinery suppliers, observes, “The current season has been very favourable for farming. It depends on when farmers bought their inputs and sold their crop. If the equation was right, they made a lot of money indeed.”

This spring I visited David Kennedy, whose family has owned land in Ayrshire for centuries, on the day that lorries came to haul away two-thirds of the feed barley in his barn. What remained still formed a floor-to-ceiling dune, 15ft high. It was worth building barns to store grain so he can put it onto the market at the right moment. “In the fields, I love and tend the plants as a crop,” he said. “Once the grain is in here, it’s just a commodity.” In theory, Kennedy’s grain could go anywhere in the world (though the cost of haulage means that he prefers to sell it locally).

This makes a world market in cereals, whose prices dictate what farmers receive for grain. Three years ago, wheat – taken as the indicator for cereals generally – stood at £60 a tonne; last year it touched £180 a tonne. Although this year the price has dropped back a bit, investors see an upward curve, reflected in the price of land. It has more than doubled since 2001.

According to James Laing of the property specialists Strutt and Parker’s rural division, the rise continues. “There were significant increases in land prices during the first six months of the year,” he says. “More people are looking for land than ever.” Farmland is now trading at an average of £8,600 an acre, according to the property consultants Smiths Gore. Funds for investing in agricultural land, formed by the asset-management company Schroders and others, have been quickly oversubscribed. No wonder. Laing believes land prices could have 30% more to go, owing to the weak pound and more food inflation expected towards the end of the year.

There is nobody better to explain the way markets have moved than Alistair Cooper in Dorset. Six years ago, he left his job at Morgan Stanley to farm the 2,500 acres he’d bought at the end of the idyllic Sydling Brook valley. While I sit in his office, the door open to the sunshine, his three-year-old son cycles by, shouting out to check he’s there. Cooper waves back; this beats 16-hour days in Canary Wharf and Hong Kong. But he didn’t leave his analytical skills at home when he bought Sydling Brook Farm. According to Justin Marking of the buyer’s agent Prime Purchase, land of this type was fetching £2,500 an acre in 2001. His investment would therefore have been over £6m. The fact that farmland has “nearly trebled” in value since is “not necessarily relevant if you’re not going to sell it”. Still, at a time when stock markets have fallen, as an investment it looks exceptionally smart. But then farmland had to go up, didn’t it?

The background is people. Babies in particular: so many of them are popping into the world that its population is growing by the equivalent of Germany’s each year. By the mid-21st century it will have jumped from the present 6.5 billion to more than 9 billion. You don’t have to be a genius to see that the demand for food is going to grow. “Global wealth has also grown,” says Cooper. “There is an almost direct correlation between income growth and meat consumption.” In the 1980s, each person in China was consuming, on average, 20kg of meat a year; 20 years later, that has more than doubled to over 50kg. With a population of 1.3 billion, that is a lot of meat. Animals need more land than crops. Every gram of meat requires about 10 times that amount of vegetable protein to produce it (less with chickens, more with beef). This has had a magical effect on the grain mountains that used to be the bane of agricultural policy in Europe. As though in a fairy story, they have vanished.

As demand has gone up, productivity has fallen. Look outside the window. It’s called the weather. Farmers, for all their computerised tractors, are at its mercy. Last July was a washout in the UK: Tewkesbury flooded and organic potato crops rotted in the fields. But arable farmers could still bring in a harvest the next month. That was more than could be said of their Australian counterparts, then suffering one of the worst droughts in history. This year it has rained in Australia, and generally the world outlook for the 2008 harvest is better than 2007.Experts are predicting a small surplus of grain. After a few years of good weather (if we get them), stocks could be replenished. Even so, it seems world grain production may have peaked; certainly there is no obvious way it can keep up with growing demand. “Rice is a massive issue,” says Cooper. “Then there is this ridiculous thing about biofuels. In the US, more land is growing biofuel than the whole of the UK agricultural area, despite the fact that the net impact of biofuel on the environment is negative.”

And on top of everything else, oil. Arguably, fossil fuels were what mucked up the climate in the first place, creating some of the shortages that have driven up the price of food. Now oil is having another effect. Wheat doesn’t have to be milled into flour: it can also be baled up and burnt in a biomass generator or stove, making it an alternative energy source. Therefore, according to Sir Ben Gill, Kendall’s predecessor as NFU president, the price of wheat inevitably follows that of oil. “Wheat produces about 18 gigajoules of energy per dry-matter tonne. If it becomes too expensive to heat your house with oil, you can burn wheat instead. Energy content has begun to underpin the value of everything.” Therefore, the price of food inexorably follows that of fuel. It looks as though high wheat prices are here to stay.

So what most of us experience as misery at the petrol pumps is good for farmers, right? Wrong.

To David Kennedy, it’s horrific. Agriculture, which accounts for 9% of Britain’s carbon footprint, is a big consumer of oil. When you’re topping up your car, spare a thought for the farmer who has to spend £750 each time he fills the combine. The specially rated “red” diesel used in farming may only cost about 70p a litre, but a tractor does a mere four miles to the gallon.

And oil as fuel is only the start of it. Nitrogen, used in fertiliser, is made from fossil fuels (principally gas) and has more than doubled in price. Phosphate, another fertiliser ingredient, has gone up even more (world demand coupled with mines flooding in China). “Last year my bill for autumn fertiliser was £160 a tonne,” laments Kennedy. “This year it will be £705 – if they can supply it.” By nature a practical man, he survived the past, low-price decade by paring costs: he buys old tractors because he can fettle them himself, and even made his own plant for turning chip oil into bio-diesel out of some plastic vats and an old chiller unit (rewired to become a heater). But the cost of pesticides, fungicide, seed and the metal needed to repair ploughs is beyond his control. And they have all gone up. Meanwhile, credit has all but evaporated. So even if the wheat price stays high, it won’t guarantee profits.

Some farmers are keeping very quiet about their performance last year. They are the ones who made the wrong call, selling their wheat crop in advance for £80 a tonne, not realising the price was about to take off; they bought their fertiliser late, at the new, higher price, and if their crop under-performed, they may even have suffered the further pain of having to buy in very expensive wheat in order to fulfil their contracts. The fact that Kennedy got it right last year doesn’t mean that he will necessarily do so again. “When inputs are low, the risks are low. But risks are getting way high. Instead of risking a loss of £20 an acre you could lose a hell of a lot more – for a meagre return. Will it pay to put any crop in the ground next year? I don’t know. It’s gone bonkers; I don’t know where it’s going to end.”

Alistair Cooper could be forgiven for feeling smug. The whole of his 2,500 acres is being converted to organic production. “My predecessor wasn’t inclined in that direction. He wanted to make the fields as big as possible. Since he owned the local agricultural machinery dealership, he wanted to get the biggest bits of kit that he could onto the land. He ploughed up every bit of downland he could get a tractor onto. He was paid by the government to do it.” Cooper, by contrast, has received grants to plant 10 miles of new hedges, making the fields smaller. Around 175 acres have been allowed to revert back to traditional downland.

Although Defra makes grants to help farmers turn organic – a hangover from the time when the priority was to reduce food production, not maximise it – the two-year conversion period is still costly. During it, the farmer has to forgo conventional fertilisers and sprays, without being able to charge the organic premium that makes up for a lower yield. But to the City boy – surprise, surprise – there’s an economic rationale. Organic agriculture is much less dependent than conventional farming on the inputs that are going sky-high.

“A conventional arable crop is sprayed eight or nine times. Each time, the tractor has to be taken out, using fuel. Wheat prices may be high, but for many farmers, profitability has not gone through the roof. By contrast, organic farmers aren’t slaves to fossil fuels. We use no fertiliser and little spray. Fertility is built up through clover and other plants that fix nitrogen. Instead of pesticides, we kill weeds by breaking up growing plants with a plough. Our yields are considerably less than conventional farmers – down by about 40%. But inputs are also less, and organic wheat trades about double the conventional price: around £290 per tonne.” It isn’t only humans who eat grain. One reason for the high price of organic wheat is that most organic farmers in Britain rear beef, sheep, pigs or chickens; they need organic grain for their feed, and not many farmers are growing it.

Whereas Kendall, in Bedfordshire, grows only crops such as wheat, oil-seed rape and beans, Cooper, in Dorset, runs the kind of mixed farm that Thomas Hardy would have recognised. Again, the economics of it stack up.

“Generally, if you’re in pigs, you can’t predict what your profitability is going to be. Nobody realised pig feed was going to double in price in the last couple of years. A conventional beef farmer doing barley bulls, which are fattened on grain rather than grass, has actually got margin compression. We have traditional British breeds of animal which we can feed from what we produce on this farm.”

We rumble around the farm in Cooper’s Land Cruiser – fields of sturdy triticale (a wheat-rye hybrid that needs little nitrogen), squeaky saddleback piglets, Poll Dorset sheep (in season all year round) and Lleyns (a prolific Welsh breed) that look like marshmallows on legs against the rich grass. “What people call organic,” reflects Cooper, “I think of as being traditional mixed farming as practised for thousands of years. The aberration is the fossil-fuel led farming of the last 40 or 50 years.”

Even technology-driven farmers like Kendall seem to be edging in Cooper’s direction. It’s simple economics. Everyone wants to reduce inputs, which means cutting waste (one of the virtues of Kendall’s new tractor is that it will save fuel). Or possibly making more use of it.

“We had a big heap of human waste delivered to a farm two miles away. It’s high in phosphate. They used to dump it at sea. We still get phone calls from the general public complaining about it, but surely it’s better than it going into the sea – and probably better than getting phosphate from mines in China.” Listeners to The Archers will know that anaerobic digesters, which turn animal waste into methane to make renewable electricity, leaving a residue of stable waste to go back onto the fields, are catching on. There are several hundred in Germany already.

Kendall is thinking differently: free-range chickens. “Battery cages will be banned in 2012, we’ve got a good local market and I need the chicken shit.” Like Cooper, he will grow more rotational crops to save fertiliser. However, he has no intention of cutting back on pesticides and fungicides: “Having spent so much money on fuel, I want to produce the best possible crops, and not see my effort wasted.”

Sydling Brook Farm in Dorset shares one characteristic with the flat southeastern tip of Kent: chalk. In both areas it is covered by only a scratching of topsoil. To Cooper, this scant land has compensations: it drains easily, and there is less difference in outputs between good and poor land under the organic system than there is in conventional farming. At Thanet Earth, where a 220-acre farm that had previously scraped a living growing cauliflowers is being re-contoured by machines that would make Peter Kendall’s eyes spin, the white skeleton barely hidden beneath the skin of soil is of no consequence: the crop won’t grow in the soil at all. Thanet Earth (cost, £80m) is in the act of becoming Britain’s biggest complex of greenhouses, covering an area equivalent to 72 football pitches. Just one of the seven glasshouses will produce 2.5m tomatoes every week of the year. The first ones will be on supermarket shelves by December.

In the world of tomatoes, peppers and cucumbers, this is state-of-the-art. They grow on shelves at waist level. The tall, airy spaces are the perfect height for tomatoes, which race up strings that are then moved sideways every week so that the tops of the vines, where the fruit forms, always remain within reach. The roots never see anything so risky as earth. They coil down into an inert fibrous material that exists merely to hold them while they absorb a precisely calculated balance of nutrients. The system is called hydroponics. Performance, feed and lighting are all controlled by computer.

Thanet Earth is at the opposite extreme from Alistair Cooper’s organics. Steve McVickers, the managing director, and Rob James, the technical manager, employed by the vegetable-marketing company Fresca, which manages the project, scoff at the idea that an organic tomato, pepper or cucumber could in any way be better than the unblemished perfection that will come from Thanet Earth. All they see in the organic system is inefficiency, which produces an unpredictable, inferior result. Some of us may find romance in the vagaries of unforced nature; we may even think –perhaps irrationally – that the taste will be better. McVickers and James don’t. Taste equals variety plus nutrients. They see hard times ahead for small organic growers, and poorly resourced conventional ones too. The cost of oil to heat the greenhouses in winter will force them to contract their season.

It is the beginning of the end, in their view. Thanet Earth, by contrast, may be alarmingly high-tech, but it is also unexpectedly green. The key to the site is a gas-powered 35-megawatts-per-hour electricity generator, capable of powering a small town. This will be connected to the power line that, en route to Ramsgate, strides over the site by a substation, now being built at a cost of £5m. Via the substation, electricity will be sold to the national grid. A by-product of the electricity generation will be heat. This will be channelled into a holding tank and then around the greenhouses, effectively free of charge. You might imagine that this industrial-scale operation would leave a carbon footprint the size of Wales, but no: it is the old-fashioned organic greenhouses that commit the sin of emissions. Thanet Earth’s CO2 is pumped back into the greenhouses: the plants need it to grow.

Water hasn’t as yet become an environmental issue on a par with animal welfare, but Peter Kendall believes it’s only a matter of time. The salad industry in parts of Spain – there are so many polytunnels around Marbella that it has sometimes been called the Costa del Condom – relied on over-abstracting water from aquifers: “They’ve been mining water,” says Kendall. “Now they’re having to move on to Morocco. Around Beijing they’re dropping the water table by a metre a year. It is such an environmental issue that the public will want information about water put on products.” At Thanet Earth, the enormous roofs are used to harvest the rain that falls; they are now scooping out great white bowls to serve as reservoirs. The development should be self-sufficient in water during the summer, only tapping into the mains when household demand is low, over the winter.

The growers behind Thanet Earth come from Holland. By Rotterdam standards, Thanet Earth doesn’t constitute a particularly big area under glass. But it lies on the right side of the Channel for British consumers. The Dutch have noted the growing anxiety about food miles. And as oil heads for $200 a barrel, it makes sense to shorten the food chain. On the Yorkshire Wolds, blonde-haired Rachel Scholes declares: “It’s vital to be able to say you’re UK-produced.” Scholes, 41, is the second generation of a potato-growing family in Driffield. Her father, John Scholes, started with a 10-acre farm in 1967; with her 34-year-old brother Richard, Scholes now farms over 5,000 acres, handling 28,870 tonnes of potatoes a year. Times are good in potatoes: following last year’s dismal harvest, prices are high for the first of this year’s crop. But the Scholeses are extending their range to flageolet beans, chicory and squash.

You can’t help feeling that flageolets are a bit fancy for Yorkshire, but hear Rachel Scholes talk about them: “They’re a beautiful green colour. Pop one in your mouth and you think it tastes like a bean, then you get this after-shot of pea – it’s the most exquisite taste.” They are the only flageolet bean growers in Britain: look for frozen packs in the northern supermarket chain Booths. The point is, flageolet beans are now local.

In Suffolk, Marybelle milk has achieved the seemingly impossible feat of capturing its own local market so successfully that even the big supermarket chains are stocking it. “When we first started the dairy six years ago, low food miles was a bit of a joke,” recalls David Strachan. “Now everybody is wanting it.”

The satellite navigation system guided me down an apparently closed-off lane, which got ever narrower, until eventually debouching onto the concrete apron of Strachan’s immaculate sheds. This is a state-of-the-art bottling and processing plant. This summer, Marybelle will be launching the first biodegradable and compostable milk carton, which could one day replace the discarded plastic milk cartons that form a large percentage of Britain’s landfill. The name Marybelle comes from Telstar Mary, the best milker from the Holstein herd that he sold last year to pay off debt, and Duncan Belle, the super-performing Canadian cow whose blood is in much of his remaining Jersey herd.

Now Strachan, helped by his wife, Colette, and sons, James and George, may have been working “18 hours a day, seven days a week, 365 days a year” since Marybelle was launched; but there’s something to show for it. Although East Anglia is not natural dairy country, or very populous, the Suffolk Riviera, with its upscale resorts and restaurants, provides a hungry market for top-of-the-range local produce. Meals at the Swan at Southwold, the Plough and Sail at Snape or in the better Aldeburgh hotels are rounded off with a choice from the 42 flavours of Marybelle ice cream. They make yoghurt and creme fraiche too; responding to demand, Strachan is thinking about butter.

Suffolk is not unduly troubled with badgers, unlike western England, where dozens may be seen at a time, with potentially devastating consequences for spreading bovine TB. (“The prime minister chastises us about wasting food,” fulminates Peter Kendall. “What about the 28,000 cattle destroyed [in 2007] because of TB?” This year the Defra supremo, Hilary Benn, rejected the scientific advice that recommended a cull, as has successfully been implemented in Ireland.) Even so, this most market-facing of dairymen keeps worrying.

“When we started to plan Marybelle eight years ago, milk stood at 26.5p a litre. It fell to 16.5p last year,” Strachan says. Milk has gone up to 25p since then, but the margin has been wiped out by the rise in feed and fuel costs. “Dairy farmers are really worse off than before we had the price increase.”

The EU quota system was designed to restrain milk production, but it hasn’t been needed in recent years. “A lot of our farming friends around here have stopped,” Strachan continues. “We have invested, but most dairy farmers have made no investment for five or six years. A new milking parlour costs £250,000.”

Dairying is being starved of the ability to renew itself. According to the NFU, 48 dairy farmers are leaving the industry every month.

As Chris Evans of the Agricultural Engineers Association puts it, “They’re looking at the price of land, and want to get out before Gordon Brown taxes it.” Dairying has yet to prove that it is the future to Strachan, for all his investment. “Farmers in their fifties and older borrow money to see themselves through in the hope it’s going to come back. But there is nothing to attract the young blood. My boys have given up well-paid jobs to work at Marybelle. They’ll give it two years.”

Strachan’s case shows that it is too early for farming as a whole to start popping the champagne corks. But there are – or have been this year – pockets of success, and above all a new mood. Food is a precious resource, and people riot when they don’t get enough of it. Within the past two years there have been food riots in Haiti, Mexico and West Bengal; many grain-growing countries imposed export restrictions and no country except the United States allowed the export of rice. Not so long ago, Gordon Brown wanted to encourage Africa to export food to countries like the UK in order to promote growth through world trade, but the risk in expecting the rest of the world to feed us is now obvious. “When I first raised the subject of food security in 2003,” remembers Sir Ben Gill, “Margaret Beckett, then head of Defra, briefed the press that I was stupid. The reality is that it’s here.” As Sir James Scott, who farms more than 2,000 acres at Rotherfield Park in Hampshire, puts it, “There is a different political and psychological mood around food and farming now. People are coming to feel more valued. Finally the importance of farming is being recognised as a producer of food.”

Also, farming has changed. Whether the farmer is driving a satellite-guided combine, farming organically, pioneering strange vegetables or taking on the supermarkets over milk, it is now a clever business. With their average age knocking 60, a lot of farmers are anxious to quit. They don’t have the skills to keep up. About 4% of them did leave last year. Has this opened up a gap for keen young people with ideas and a bit of capital to get in? James Laing is not encouraging.

“You wouldn’t want to start unless you had 1,000 acres and that will cost you £8m. Then you’ll need working capital to set yourself up.” When I suggest £5m, he laughs, “£50m, more likely,” adding: “A smallholding is going to bankrupt you.” Rachel Scholes thinks there are openings for people with “innovative ideas for niche products, like green carrots or orange raspberries. The more interesting food looks, the more likely people are to pick it off the shelf”.

Peter Kendall explains the routes you could take. “You might start off working for someone, share-farming or working in partnership. Lots of people have grass to rent out. Existing farmers have a head start because they’ve got machinery, but there’s a great market for free-range eggs.” An even surer point of entry would be through agricultural science. Whatever the future of farming for food, one thing is certain: as fossil fuels are phased out, plants will have to provide some of the energy, and even plastics, that we now get from oil. They will have to multitask. Someone who can devise a crop yielding not only protein (to eat) but ethanol (to drive the car) and fibre (to make your clothes) while requiring less nitrogen, phosphate and water to grow, will make a fortune. Oh, and save the world too.

Clive Aslet is editor-at-large of Country Life