Text Mining

DPLA’s API provides programmatic search and access to every item in the DPLA catalog. Use the API to power an app, to wire DPLA into your portal, or to retrieve data.

Multiple collections of digitized primary sources related to southern history, literature, and culture. Some collections offer plain-text downloads in their entirety: The Church in the Southern Black Community, First-Person Narratives of the American South, Library of Southern Literature, North American Slave Narratives.

Google nGram viewer allows you to graph word frequency across the corpus of Google Books.

Nearly 14 million books from the HathiTrust Library are currently available for analysis, offering various levels of immediate access. Check the Hathi Trust Research Center tab on this guide for more information to help you get started.

Project Gutenberg hosts over 50k ebooks, most of which are older books in the public domain. If you want to download more than about 100 books/day, use one of the mirror sites listed from the link above.

Contact the Library to initiate TDM requests.

Contact the Library to initiate TDM requests.

Full-text and metadata bulk downloads for open access scholarship in Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance, Statistics, Electrical Engineering and Systems Science, and Economics.

Peer-reviewed Biomed Central articles are available for text and data mining through the RESTful API.

Contact the Library to initiate TDM requests.

Article full text, metadata and citations may be crawled for the purpose of an electronic analysis without special permission or registration, on the condition that it is non-commercial. Any reuse of content for this purpose shall be subject to the restrictions set out in this table.

CORE provides a central API to access full content from tens of thousands of openly available scientific publications from thousands of OA repositories. They also provide full datasets by request.

Provides access to metadata records with CrossRef DOIs, covering about 75 million scholarly works from around 5000 publishers.

Contact the Library to initiate TDM requests.

Researchers can text mine UCM-accessible journals and books published by Elsevier on the ScienceDirect full-text platform. Sign up for a developer account to use the Elsevier APIs for non-commercial purposes, and make sure to query the API from UC Merced IPs to ensure full access. For more information, see their Text Mining documentation.

The IEEE Xplore Metadata API provides access to metadata for millions of documents available in the IEEE Xplore Digital Library including IEEE journals, conferences, books / ebooks, courses and standards; accessible through the IEEE Xplore API Portal .

Data for Research allows you to download word frequencies, citations, key terms, and ngrams for up to 25,000 JSTOR articles at a time, or to easily submit requests for larger sets of articles. See also:

• JSTOR DfR in GitHub - A number of Python and R packages to work with JSTOR DfR data.
• JSTOR's Text Analyzer, a reverse search engine that analyzes documents that you upload (your own, or other articles) to find related materials in JSTOR.
• Public domain and OA datasets include full OCR text from early journals and current academic press open access titles

Contact the Library to initiate TDM requests.

Python tool for downloading/updating/maintaining a repository of all PLOS XML article files. Use this program to download all PLOS XML article files instead of doing web scraping. See also: PLOS APIs to query content from the seven open-access peer-reviewed journals from the Public Library of Science using any of the twenty-three terms in the PLOS Search.

Members of subscribing institutions have permission to mine journal content. Before carrying out data or text mining, follow instructions here, Data sharing and mining, for contacting the Royal Society of Chemistry.

Springer Nature offers a variety of TDM tools such as metadata and fulltext APIs, for both open access and subscribed resources. For more information, visit Springer's Text and Data Mining Policy.

Contact the Library to initiate TDM requests.

Wiley supports TDM on Wiley content. Access to content for TDM purposes takes place through the Crossref Text API. Visit Text and Data Mining for Researchers for details.

A shared commons of data and tools for accessing data from the United States government.

The Caselaw Access Project (“CAP”) expands public access to U.S. law, and contains over 360 years (going back to 1658) of published U.S. court decisions, digitized from the collection of the Harvard Law Library.

Sunlight Foundation provides APIs to a variety government information sources including: Congress API v3: information on legislators, districts, committees, bills, votes, as well as real-time notice of hearings, floor activity and upcoming bills. Open States API: Information on the legislators and activities of all 50 state legislatures, Washington, D.C. and Puerto Rico. Real-Time Federal Campaign Finance API: A JSON and CSV API that delivers up-to-the-minute campaign finance information on federal candidates, committees, PACs and other groups that file electronically with the Federal Election Commission.

Opinions, docket files, and more from 420 courts.

Bulk data downloads of major US Government publications including Congressional Bills, Commerce Business Daily, Federal Register, Public Papers of the Presidents of the United States, Supreme Court Decisions 1937-1975 (FLITE) and more.

CoRD provides links to and descriptions of a large number of corpora, subcorpora and databases. (University of Helsinki)

The LDC collects language data from both written texts and transcriptions of speech, in various languages, to support corpus linguistics. Visit Accessing the Linguistic Data Consortium for collection and account information.

15 million words of American English automatically annotated for logical structure, word and sentence boundaries, part of speech (multiple tag sets), shallow parse (noun and verb chunks), and named entities.

The Scottish Corpora project has created large electronic corpora (over 4.5 million words) of written and spoken texts for the languages of Scotland. See also the Helsinki Corpus of Older Scots (1450-1700) and the Corpus of Modern Scottish Writing (1700-1945).