It is difficult for patients to perceive the quality of prescription drugs even after they consume them, because they lack medical knowledge and because symptom alleviation depends on comorbidities, diet, and other factors. Additionally, multiple factors, ranging from active pharmaceutical ingredients (API) and purity to transportation and storage conditions, affect drug quality. After a pharmaceutical product is approved for the market, ensuring quality in production and distribution is crucial.

This is why the U.S. Food and Drug Administration routinely inspects domestic drug manufacturing plants for compliance with good manufacturing practice. In theory, all drugs approved for use in the United States, whether made domestically or overseas, must comply with applicable federal regulations. But it is more difficult to conduct the good manufacturing practice inspection abroad, where roughly 80 percent of APIs and 40 percent of finished drugs are made. Because the FDA lacks legal jurisdiction outside the U.S., it can only send warning letters, issue import alerts, and deny market access for foreign plants that are found to be out of compliance.

In the early years of this century, several high-profile episodes of contaminated imports resulted in substantial numbers of deaths in the United States, leading the FDA to strengthen its quality control program in China, India, and many other countries. Even after these steps, however, the FDA mainly inspects wholesale imports. "While the Federal Food, Drug, and Cosmetic Act (FFDCA) prohibits the importation of unapproved drugs into the U.S., including retail purchases by individuals who shop on the internet, the FDA does not vigilantly enforce the ban. (1) Thus it is possible that some low-quality pharmaceutical products are still reaching U.S. consumers as a result of online retail sales.

My collaborators and I have studied the trade-offs that are associated with direct-to-consumer pharmaceutical sales by internet sellers, studying in particular the quality variation across drug providers and the evolving access that U.S. buyers have to offshore sellers.

Certification and Quality Variation

Four to six percent of U.S. residents order prescription drugs from online pharmacies. (2) Some are foreign pharmacies that may not meet FDA standards. The National Association of Boards of Pharmacy (NABP) reviewed 7,430 internet pharmacies in December 2010 and found that 96 percent were not in compliance with U.S. state and federal laws and/or NABP patient safety and pharmacy practice standards. (3) Among non-compliers, 34 percent had server locations in a foreign country, 27 percent had a physical address outside the U.S., 56 percent did not provide any physical address, 84 percent did not require valid prescriptions, 62 percent issued prescriptions via online consultation, 50 percent offered foreign or non-FDA-approved drugs, and 83 percent did not offer medical consultation. These findings suggest that the rise of internet-marketed pharmaceuticals has introduced new concerns about drug quality.

The NABP, which emphasizes that consumer importation of drugs violates the FFDCA, certifies U.S. web-based pharmacies that comply with laws in both the state of their business operation and the states to which they ship. As of February 29, 2012, the NABP had certified 30 online pharmacies. Twelve of these were run by large pharmacy-benefits management companies open to members only; others are the online branches of national chain pharmacies such as CVS.com and Walgreens. com, and large online-only pharmacies such as drugstore.com.

Another certification agency, LegitScript.com, has a similar focus on U.S. websites. It was endorsed by the NABP to screen pharmacy websites after 2011. As of November 2016, LegitScript was monitoring over 80,000 Internet pharmacies. It estimated that between 30,000 and 35,000 were actively selling prescription drugs at any one time. Among active websites, 96 percent did not satisfy LegitScript's certification criteria and therefore were not fully compliant with U.S. laws and regulations. (4) NABP endorses the use of LegitScript by domain name registrars to assist in identifying illegally operating websites; I therefore consider websites certified by either agency as NABP-certified.

The other two private certifiers--PharmacyChecker.com and the Canadian International Pharmacy Association (CIPA)--are fundamentally different. CIPA is a trade association of Canadian pharmacies and only certifies Canadian websites that comply with Canadian laws, while PharmacyChecker covers the U.S., Canada, and many other countries. Similar to NABP, PharmacyChecker also charges fees for an approved website to be listed on PharmacyChecker.com beyond a short initial period. (5)As of March 9, 2012, at about the time my collaborators and I were studying this issue, PharmacyChecker had approved 73 foreign websites and 51 U.S. websites. Because PharmacyChecker is unwilling to share its complete list of approvals, it is impossible to conduct a full comparison between approvals by PharmacyChecker and those by the NABP, LegitScript, or the CIPA. Among the four certification agencies, PharmacyChecker is the only one that provides head-to-head drug price comparisons across online pharmacies.

To investigate whether drug quality differs between certified and uncertified online sellers, Roger Bate, Aparna Mathur, and I obtained 365 samples of five popular brand-name prescription drugs from three tiers of online pharmacies: NABP-certified websites (tier A), PharmacyChecker/CIPA-certified websites (tier B), and websites that were not certified by any of the four certifiers (tier C). (6) We then compared all the testable samples (328) with authentic versions, using the Raman spectrometer. (7) There was zero failure in tier A and tier B samples, but eight tier C samples of Viagra failed the authenticity test. All other testable drugs that were purchased from Tier C passed. This finding validates concerns that using uncertified online pharmacies may be risky, but the lack of failure among tier B pharmacies also suggests that not all foreign pharmacies are rogue.

It is important to note that there can be price differences in addition to quality differences. In our audit test, although tier A and tier B samples exhibited similar quality, tier B samples were 49.2 percent cheaper than tier A samples after controlling for other factors. Samples from tier C websites were 54.8 percent cheaper than those from tier A websites. Importantly, these differences were driven by non-Viagra drugs, all of which passed the authenticity test. In contrast, the failing samples of Viagra were cheaper than the passing samples, but there was no significant price difference across tiers once we conditioned on testability and authenticity.

The large price difference between tier A and the other two tiers highlights price variations in the international market of prescription drugs. Because many non-US. countries are willing to impose price regulations on prescription drugs, the same drug could be much cheaper outside the U.S., even if the drug was patented by a U.S. manufacturer. For example, a 2005 study estimated that Canadian prices for the 100 top-selling brand-name drugs were on average 43 percent below U.S. prices for the same drugs. (8) As a result, saving money is one of the leading reasons to buy prescription drugs online, despite quality uncertainty. (9)

Access to Internet Pharmacies

It is easy for shoppers to find "cheap" pharmacies online. Internet shopping allows U.S. consumers to access low-priced drugs, while also allowing rogue pharmacies to take advantage of gullible consumers. This trade-off has been recognized by the platforms that permit consumer search. As online pharmacies expanded, Google contracted with PharmacyChecker to filter websites listed in its sponsored search results. However, a Department of Justice investigation found that Google was allowing unapproved pharmacies to purchase sponsored links and target U.S. consumers. In February 2010, Google started to ban pharmacies not certified by the NABP from sponsored ads targeting U.S. consumers, and to block pharmacies not certified by the CIPA from sponsored ads targeting Canadian consumers. Other search engines followed suit. In August 2011, Google settled with DOJ and agreed to forfeit $500 million in ad revenues.

How does the ban of sponsored ads affect consumer search and click behavior concerning online prescription drugs ? Matthew Chesnes, Weijia (Daisy) Dai and I apply synthetic control and difference-in-differences (DID) to corn-Score click-through data from 1 million U.S. households. (10) The monthly click-through data, ranging from September 2010 to September 2012, track the number of searches and searchers for a query and the number of organic and sponsored clicks on each website that result from the query. To be comprehensive, we started with more than 8,000 health-related queries and narrowed down to 528 queries that either accounted for the majority of click volume or were most likely leading to pharmacy websites. Searches using drug and pharmacy queries generated 97 percent of the traffic on pharmacy websites.

We sorted online pharmacies into the same tiers: tier A for NABP-certified websites, tier B for PharmacyChecker or CIPA-certified websites, and tier C for websites not certified by any of the four agencies. By definition, tier A pharmacies are not subject to the ban on sponsored listings and therefore not the subjects of the study. For each tier B or tier C pharmacy website, we constructed a control group sample using clicks on health-related non-pharmacy websites following drug and pharmacy queries. The synthetic control sample is chosen such that organic clicks reflect the same underlying trend in consumer interests as the treated website, but are not directly affected by the search engine ban.

We found the ban to have heterogeneous effects on pharmacy websites. As shown in the left panel of the figure below, tier B websites experienced a large increase in organic clicks--clicks on the search results based on relevance to the search term rather than advertiser's payment--making up for roughly two-thirds of their loss of sponsored clicks. These results suggest that the ban increased the search cost for tier B sites, but that some consumers overcome the search cost by switching from sponsored to organic links. Indirectly, these results also suggest that a sponsored search was likely effective for tier B sites before the ban, though we may not causally attribute all sponsored clicks before the ban to the effect of the advertising. In contrast, in the right panel of the figure, tier C pharmacies barely made up any of the loss in sponsored clicks, which can be explained by rising consumer concerns about the quality of drugs sold on uncertified websites after the ban, thanks to related media exposure and government advocacy.

The differential effects on tier B and tier C sites suggest that consumers may have become more cautious about which websites they buy from and more likely to use information from third-party certifiers. Furthermore, the increase in organic clicks on tier B pharmacies tends to come from queries that target discount pharmacies, and the most significant increase in organic links is for the drugs that treat chronic conditions.

To summarize, while there is stringent regulation of drug quality in the U.S., personal imports still expose U.S. consumers to the potential risk of unsafe and low-quality drugs. This risk has been addressed by private certification and enforcement action on a major search engine, but the cost of these actions is that U.S. consumers may face higher search costs and have less access to lower international prices.

A similar price-quality trade-off exists in developing countries and affects a much larger population. Bate, Mathur, and I focused on eight drug types on the WHO-approved medicine list and obtained 899 drug samples from seventeen low- and median-income countries. (11) We tested for visual appearance and disintegration, and analyzed their ingredients by chromatography and spectrometry. Fifteen percent of the samples failed at least one test and failing drugs were priced 13.6-18.7 percent lower than non-failing drugs after controlling for local factors, but the signaling effect of price is far from complete, especially for non-innovator brands.

In a subsequent study, we assessed basic quality of 1,437 samples of Ciprofloxacin, an antibiotic that is used to treat many bacterial infections, from 18 low- to middle-income countries. (12) Following the Global Pharma Health Fund e.V. Minilab[R] protocol, we found that 9.9 percent of the samples had less than 80 percent of the correct API, of which 41.5 percent were entirely falsified, containing zero API, and the rest were substandard, with measurable but insufficient API. (13) Although substandard drugs are on average cheaper than passing generics in the same city, the average price of falsified drugs is not significantly different from that of passing drugs. These patterns suggest that careful consumers may suspect a drug is substandard before purchase, but it is more difficult to identify falsified fied drugs ex ante, since they mimic the price and packaging of high-quality, locally registered products.

Ginger Zhe Jin received her PhD from UCLA in 2000 and is currently a professor of economics at the University of Maryland, College Park. She was on leave to the Federal Trade Commission in 2015-17, and served as director of the FTC's Bureau of Economics from January 2016 to July 2017. She has been an NBER research associate since 2012, and is a member of the bureau's Industrial Organization Program.

Most of Jin's research focuses on information asymmetry among economic agents and how to provide information to overcome this problem. She has studied restaurant food safety, health insurance, prescription drugs, e-commerce, online reviews, regulatory inspection, scientific innovation, air quality, blood donation, intrafamilial interaction, data security, and consumer protection. Her research has been published in economics, management, and marketing journals, and has been supported by the National Science Foundation, the Net Institute, and the Alfred P. Sloan Foundation. She is currently co-editor of the Journal of Economics and Management Strategy, associate editor of RAND Journal of Economics, and guest senior editor for a special issue of Marketing Science on consumer protection.

(1) The FDA defines personal drug imports as those that represent a reasonable risk and are intended for personal use of no more than a three-month supply. An agency handbook on drug-product control states that when determining the legality of personal shipments, "FDA personnel may use their discretion to allow entry of shipments of violative FDA regulated products when the quantity and purpose are clearly for personal use, and the product does not present an unreasonable risk to the user."

(2) G. Orizio, A.Merla, P. Schulz, and U. Gelatti, "Quality of Online Pharmacies and Websites Selling Prescription Drugs: A Systematic Review," Journal of Medical Internet Research, 13(3), 2011. https://www.ncbi.nlm.nih.gov/pubmed/219 65220

(3) National Association of Boards of Pharmacy. "Internet Drug Outlet Identification Program Progress Report for State and Federal Regulators" January 2011.

(4) LegitScript's certification criteria include a valid license with local US jurisdictions, a valid registration with the US Drug Enforcement Administration (DEA) if dispensing controlled substances, valid contact information, a valid domain name registration, a valid prescription, dispensing only FDA-approved drugs, and protecting user privacy according to the HIPAA Privacy Rule (45 CRF164). There are more LegitScript-certified websites than NABP-certified websites, probably because the NABP requires interested websites to apply and pay verification fees while LegitScript's approval is free and does not require website application.

(5) Certification requirements on Pharmacy Checker include the stipulation that any approved website must have a valid pharmacy license from its local pharmacy board, requires a prescription for US purchase if the FDA requires a prescription for the medication, protects consumer information, encrypts financial and personal information, and presents a valid mailing address and phone number for contact information.

(6) The prescriptions drugs were Lipitor (10 mg), Viagra (100 mg), Celebrex (200 mg), Nexium (40 mg), and Zoloft (100 mg).

(7) The samples were also cross-checked against a second lot from a separate national pharmacy chain store to verify consistency and determine method robustness. R. Bate, G.Jin, and A. Mathur, "In Whom We Trust: The Role of Certification Agencies in Online Drug Markets" NBER Working Paper No. 17955, July 2013, and Berkeley Express Journal of Economic Analysis & Policy, Contribution Tier, 14(1), 2013, pp. 111-50.

(8) B. Skinner, "Canada's Drug Price Paradox: The Unexpected Losses Caused by Government Interference in Pharmaceutical Markets," Fraser Institute Digital Publication, 2005. https://www.fraserinstitute.org/sites/default/files/CanadasDrugPriceParadox.pdf.

(9) According to C Gurau, "Pharmaceutical Marketing on the Internet: Marketing Techniques and Customer Profile," Journal of Consumer Marketing, 22(7), 2005, pp. 421-8, the most frequent reasons quoted by interviewees for buying or intending to buy online were convenience and saving money, followed by information anonymity and choice. In our own survey (R. Bate, G.Jin, and A. Mathur, 2014) of 2,522 members of RxRights, 61.54 percent purchased drugs online, mostly from foreign websites, citing cost savings as the leading reason.

(10) M. Chesnes, W.Dai, and G Jin, "Banning Foreign Pharmacies from Sponsored Search: The Online Consumer Response," NBER Working Paper No. 20088, May 2014, and Marketing Science, 36(6), 2017, pp. 879-907.

(11) R. Bate, G.Jin, and A. Mathur, "Does Price Reveal Poor- Quality Drugs? Evidence from 17 Countries," NBER Working Paper No. 16854, March 2011, and Journal of Health Economics, 30(6), 2011, pp. 1150-63.

(12) R. Bate, G.Jin, and A. Mathur, "Falsified or Substandard? Assessing Price and Non-Price Signals of Drug Quality," NBER Working Paper No. 18073, October 2012, and Journal of Economics & Management Strategy, 24(4), 2015, pp. 687-711.

(13) Substandard and falsified drugs are not necessarily counterfeits. The World Health Organization in 2010 (http://www.who.int/en/news-room/fact-sheets/detail/substandard-and-falsified-medical-products) defined counterfeit drugs by the intent to deceive, but it is extremely difficult to prove intent in practice, especially if the focus is on the intent to infringe trademark rather than the intent to provide effective medicines. As a result, deliberating on trademark infringement often diverts attention from drug quality and its public health implications. By this definition, a counterfeit drug could have zero, some, or even full content of API, if it infringes the trademark; in the meantime, substandard or falsified drugs could be produced by manufacturers that have the legal trademark and other IP rights on the drug.