People did not start drilling for buried petroleum until the middle of the 19th century, though its existence had been known for centuries. In the mid-19th century, a shortage of whale oil coincided with the invention of techniques for refining "rock oil" into lamp oil (see Abraham Gesner). Since then, demand for oil and gas has periodically surged and slumped, as have prices and exploration activity. Regardless of the market for petroleum, however, both the exploration and production of oil are expensive endeavours. Even with the most sophisticated technology, finding hidden petroleum deposits involves creative interpretation of the land. In Canada, petroleum exploration and production takes place in every province and territory, with the exception of Nunavut, which lags behind in petroleum development.

Geology and Geophysics

The Earth's surface often provides the field geologist with clues to what lies below. The hints may be obvious: seepages of oil and gas; or outcroppings of sedimentary rock or fossils exposed in a tilted layer of rock on a mountain. That layer of rock might once have been the flat bottom of an ancient sea which, through millions of years and constant movements of the Earth's crust, was folded, tilted and exposed (see Plate Tectonics).

Accumulations of oil and gas are often found in these folded or tilted rock strata. However, the rocks that may contain traps for reservoirs of oil and gas are often thousands of metres below the surface, sometimes covered by a fairly horizontal plain. Aerial photographs and topographical maps can be useful in determining the underground landscape, and the science of geophysics can be even more helpful in mapping the depths beneath the surface.

Geophysics studies physical properties of rocks (e.g., magnetism, resistivity and radioactivity). The geophysicist's tools include a magnetometer and a gravimeter, which are used to identify rock types. Both instruments were developed in the 1800s and have been improved continually since that time. Deeply buried sedimentary rocks are often very dense and contain high concentrations of iron and other materials detectable by a magnetometer. Differences in the pull of gravity of various rocks, measured by the gravimeter, can also help identify the locations of certain rock types beneath the surface.

Another geophysical tool, the seismograph, can virtually compose pictures of underground rock formations. The seismograph works on the principle of sound-wave transmission. The sound waves are generated by small charges of dynamite or through vibrating, or percussion methods. As they travel downwards through layers of different types of rocks, the sound waves are reflected back to the surface. Differences in wave intensity and travel time provide information about the different types and structures of rocks through which the waves travel. At the surface, the sound waves are recorded by geophones, sensitive recording devices connected to computer equipment. Computers produce the "pictures" (i.e., seismographs) in which the geophysicist can "see" the structural composition of the rocks far below.

Exploration

The only way to prove what lies in buried rocks is to drill a well. Even with modern geology and geophysical techniques, drilling remains risky. An exploratory or "wildcat" well in an area that has not been drilled before faces high odds against success: approximately one of seven to 10 exploratory wells finds commercial accumulations of oil or gas. To be commercially viable, a well must be able to produce enough oil or gas to justify the costs of drilling and placing it on production. In wildcat areas the first exploratory wells are often drilled as tests, and they are not expected to yield oil or gas. Such wells, however, produce valuable information about the nature of the rocks, and their oil and gas potential through the analysis of core samples, rock cuttings and data gathered from down-hole surveys.

If exploratory wells establish the presence of producible quantities of oil or gas, "development" wells are drilled to define the size and extent of the field. In development drilling the odds for success are higher: about six or seven successful wells for every 10 drilled. But the element of risk is still present. There may not be enough oil or gas to make the site commercially feasible or the technology required to produce oil or gas may be too expensive.

The exploration industry in Canada has gone through ups and downs that have little to do with the amount of oil or gas left to be found and more to do with the balance between supply and demand. For example, the Norman Wells oil field in the Northwest Territories was first discovered in 1920, but it was then considered too remote to be of interest to southern Canada. A modest amount of oil was refined there to meet regional demands. It was not until the mid-1970s that decreasing supplies and increasing oil values made large-scale development at Norman Wells worthwhile.

Technology also can have an impact on exploration activity. For example, extensive exploitation of the large volumes of heavy oil in the Lloydminster area was dependent on the development of enhanced recovery techniques.

Drilling

In the early years of Canada's petroleum industry, wells were not drilled but were punched with cable tools. On a cable tool rig, a heavy bit with a chiselled edge was suspended on a line of rope or wire cable. The hole was made through the constant raising, lowering and pounding of the bit into the earth. By the late 1920s, most operations used rotary drilling equipment, which was more efficient, drilling deeper and faster.

To a large extent, drilling still involves turning a drill bit at the end of lengths of steel pipe. Each piece of pipe is about 9 m long and is added, a length at a time, by threading onto the next piece of pipe. The bit, drill collars (which add weight to the bit) and lengths of pipe are called the drilling "string." The whole string is turned by a rotating platform, the rotary table. The revolving bit cuts and grinds through rock formations, lubricated and cooled by drilling fluid commonly called drilling "mud," a mixture of water or oil, clay and chemicals.

A deep-rated drilling rig, which might be used to drill holes 5,000 m deep, is composed of much heavier, larger and stronger equipment than one used to drill shallow wells (e.g., 1,000 m deep). For offshore drilling, rigs generally are permanently mounted on barges or platforms so that they can be towed from well site to well site. Some offshore drilling rigs are mounted on specially designed ships which move under their own power.

Downhole Technologies

During recent decades the technologies used to find and produce oil have gone through a revolution. In particular, horizontal drilling and the ability to drill several horizontal laterals from a single wellbore have transformed the drilling sector. Horizontal drilling is possible because of improvements in bit design, better downhole motors and bigger rigs.

Geo-steering is one of a number of revolutionary downhole technologies developed since the 1990s. In recent years, it has been given a lift by high-impact, measurement-while-drilling tools and techniques. More importantly, the industry can now isolate many completion zones (areas from which oil and gas are produced) in horizontal wellbores. This makes reservoir fracturing (and therefore production) possible from horizontal wells that reach out for several kilometres underground.

It took rapid innovation in downhole tools to turn hydrocarbon-bearing shales and other low-permeability rocks into producing reservoirs. Coil (“coiled”) tubing is the workhorse of underground technologies. A tool that began to make inroads into industry operations around 1990, coil tubing has transformed many aspects of underground drilling and work-over operations. It refers to metal piping spooled on a large reel and used for downhole operations in wells. Coiled tubing is often used to carry out operations previously done by wirelining, which involves lowering tools into a wellbore by an electrical cable. Wireline tools transmit data about the conditions of the wellbore to the surface. The main benefit of coil tubing over wireline is that you can pump chemicals through the coil. With coil tubing you are able to push tools and chemicals into the hole, whereas wirelining relies on gravity.

Of particular importance in the context of production from shale, coil tubing can be used to fracture the well, a process where fluid is pressurized to thousands of pounds per square inch on a specific point in a well. This blasts some of the rock into rubble, permitting the flow of hydrocarbons to the wellbore.

The tool string at the bottom of the coil can range from something as simple as a jetting nozzle, for jobs involving pumping chemicals or cement through the coil, to a larger string of logging tools, depending on the operations. Coil tubing is also used for relatively inexpensive work-over operations. It is also used to perform open-hole drilling operations.

As these technologies increased in sophistication and declined in relative cost, they led to a fundamental change in field economics. Producers are now investing more money underground than above ground.

Completion

Drilling operators constantly monitor the progress of a well so they can quickly make decisions about whether to “complete” the well so it can be used for production or “abandon” it as a dry hole. Throughout the drilling operation, rock cuttings are examined for traces of hydrocarbons, and other evaluations and analyses are made. If the well is judged a dry hole, it will be plugged with cement and abandoned. However, if the tests show promise, the well will be "completed."

The first step in completion is the installation of production casing, i.e, a tubular steel pipe that is cemented in place down the length of the wellbore. After this process, the drilling rig is usually removed from the well and a truck-mounted service rig is moved into place. The production casing is perforated to allow entry of fluids and gases into the wellbore. The perforations also provide access to the producing formation for other completion activities that may be undertaken.

Formations that years ago might have been considered capable of only minimal production now achieve good production rates through completion practices that, with better prices for oil and gas or improved technology, are worth implementing. For example, fracturing, or “fracking,” is a common completion technique. In this process, materials are pumped down the well under high pressure to pop open cracks in the reservoir rock so that the oil or gas can move more freely through the formation.

Production

Because crude oil and natural gas are non-renewable resources, optimizing recovery is critical. Only about 25 per cent of the oil can be recovered from a typical reservoir by natural means or primary recovery techniques. Enhanced-recovery techniques permit production of more oil from many reservoirs. The most common enhanced-recovery method, water injection, involves injecting water into the oil-bearing formation, and the water then forces the oil toward the producing well bore. Such techniques can result in recovery rates that can exceed 80 per cent of the oil in place.

Natural gas generally flows to the surface through its own pressure; thus, a natural gas wellhead is usually composed of only a series of chokes and valves to control flow. This wellhead structure is called a "Christmas tree." Crude oil, which typically contains some natural gas or solution gases, is sometimes produced through its natural pressure, but most crude oil wells in Canada require some method of lifting or pumping the oil to the surface. Pumping equipment is known by various names, including "pump jack," "horsehead pump" and “walking beam” pump.

Transportation

The transportation of oil and gas from areas of supply to areas of demand is critical. In western Canada's oil- and gas-producing provinces, long trains of tank cars are a common sight. They carry everything from asphalt (produced from heavier crude oils and used for paving roads) to propane, butane, and other liquid and gaseous products of crude oil and natural gas. Because Canada and many areas in the United States are increasing oil production, railroads are experiencing rapid growth in the number of tank cars taking petroleum across the continent to industrial centres where it can be used in a great variety of manufacturing processes (see Chemical Products Industries). Burgeoning traffic in petroleum rattling along railways has raised serious safety concerns, most notably following the derailment in Lac-Mégantic, Qué on 6 July 2013.

Before the 1950s, railway and truck transport were the only methods of transporting oil and gas across Canada; however, since then pipelines have carried most of Canada's oil and gas production to areas of consumption. Just as the construction of railways was essential to the early development of Canada, pipelines became an integral part of Canada's industrial growth midway through the 20th century.

Refining

Gasoline, diesel fuel and jet fuel are the most common petroleum products, but the list of manufactured products includes everything from insecticides to shampoo and plastic. The processes that result in these products are often complex, but all are based on separating crude oil's various components into useful by-products, which can be in solid, liquid or gaseous forms.

Refining begins with heating crude oil past its evaporation point. In a process called distillation, the various components are vapourized and separately condensed according to their boiling points into basic hydrocarbon streams. Gasoline, kerosene, jet fuel and diesel fuel are produced from middle distillates; greases, lubricating oils, waxes and asphalt are produced from residues.

Some products must be blended and improved with chemical additives in secondary refining processes to produce finished products with desired characteristics. Refineries may also produce heating fuels, heavy industrial fuels and feedstocks for the petrochemical industry.

Regulation

The marketing of crude oil, natural gas, their products and by-products is complex. Various regulations, both federal and provincial, govern all aspects of production and sales. Although the provinces have jurisdiction over the oil and gas produced within their boundaries, the federal government has the ultimate jurisdiction over oil and gas pricing, transmission and sales, both domestic and export. Governments share the revenues of producing companies through federal and provincial taxes, and provincial royalties.

The ownership of oil and gas, particularly in offshore areas, and the right to revenue sharing and regulation often have been topics of dispute among federal and provincial governments (see Energy Policy). The National Energy Board regulates interprovincial movements of oil and gas and export allocations of gas and electricity.