Resources.data.gov has replaced Project Open Data.
On January 14, 2019, the Foundations for Evidence-Based Policymaking Act ("Evidence Act"), which includes the OPEN Government Data Act, was signed into law. The Evidence Act requires the Office of Management and Budget, the Office of Government Information Services, and the General Services Administration to develop and maintain an online repository (Resources.data.gov) of tools, best practices, and schema standards to facilitate the adoption of open data practices across the Federal Government.
Archived content of Project Open Data can be accessed at its GitHub repository. You can report missing content or provide any additional feedback via GitHub or by emailing datagov@gsa.gov.
##Frequently Asked Questions
###Questions Applicable to Contracts:
###Questions Applicable to Grants and Other Financial Assistance:
####Q1: What sort of data is covered by the open data policy? A: The purpose of the Open Data Policy is to make open and machine readable the new default for government data collection and dissemination in order to enhance availability, access and interoperability, per May 9, 2013 Executive Order released by the White House. This includes all information generated and stored by the Federal Government or for data purchased or created through Federal funding such as data collected in conjunction with program administration, scientific research, public health surveillance. There is an expectation that agencies will work to prioritize ensuring that data is open when it would be likely to fuel entrepreneurship, innovation, accountability, and scientific discovery, and improve the lives of Americans in tangible ways.
####Q2: Who within my agency is responsible for Open Data Policy decision-making? A: The senior accountable official in your agency for making those decisions is the CIO, in partnership with the data owners in your agency.
####Q3: What does “open data” mean? A: Open data is data that is publicly available data and structured in a way that enables these data to be fully discoverable and useable by end users inside and outside of government. In general, open data is consistent with the following principles: public, accessible, described, reusable, complete, timely, and managed post-release (attributes defined in the Office of Management and Budget (OMB) Memorandum M-13-13 Open Data Policy – Managing Information as an Asset.
####Q4: What does “platform independent” mean? A: Platform independent data do not require a specific program to open (e.g., a comma-separated value (.csv) spreadsheet). They are in a format that can be read and processed by a variety of software tools – including free tools.
####Q5: What does “machine readable” mean? A: Machine readable data is data structured in a format that can be understood and processed by a computer. Some file formats, such as a PDF or Word documents, are human readable and easier to read and edit. This differs significantly from machine readable formats that can be processed by a computer and parsed or organized around specific information. OMB defines “Machine Readable Format” and provides some examples in Circular A-11 Part 6, “Preparation and Submission of Strategic Plans, Annual Performance Plans, and Annual Program Performance Reports.” The circular describes machine readable format as:
a standard computer language (not English text) that can be read automatically by a web browser or computer system. (e.g.; xml). Traditional word processing documents, hypertext markup language (HTML) and portable document format (PDF) files are easily read by humans but typically are difficult for machines to interpret. Other formats such as extensible markup language (XML), [JavaScript Object Notation] (JSON), or spreadsheets with header columns that can be exported as comma separated values (CSV) are machine readable formats. It is possible to make traditional word processing documents and other formats machine readable but the documents must include enhanced structural elements.
In addition, M-13-13, “Open Data Policy – Managing Information as an Asset” encourages agencies to make data available in non-proprietary (i.e., platform independent) formats to the extent permitted by law. The preferred format for making data available is non-propriety (i.e., platform independent) because this platform allows the data to be accessed without the required use of certain proprietary software programs such as Excel. File formats that are preferred are identified in A-11 Part 6 and include JSON, XML, and CSV. Your agency CIO will be able to provide more information and subject matter expertise to help contracting officer representatives (CORs) or other program officials in understanding requirements and how to assess deliverables.
####Q6: Are all agencies approaching this data in the same way? What if my agency has a grant or contract with recipient A and we translate the data but another agency has a similar contract with recipient A and requires the recipient to provide the data in a platform-independent, machine-readable format? How does this help with consistency across the government?** A: OMB is working collaboratively across the Executive Office of the President to better understand how agencies are implementing open data and providing input and information to smooth the implementation process for grant recipients and contractors. As part of that process, we will be working across the Councils identified in Section 3 (b) of the Executive Order (EO) “Making Open Data and Machine Readable the New Default for Government Information” to better understand overall implementation efforts and further develop tools and information that provides consistency and avoids unnecessary duplication.
####Q7: Must all agency data be posted publicly? What sorts of considerations guide whether data should be posted publicly? A: No. Neither the E.O. nor policy change Agency obligations under any law. Specifically, nothing in the EO or policy compels or authorizes the disclosure of privileged information, law enforcement information, national security information, personal information, or information the disclosure of which is prohibited by law. For example, for contracts, information subject to the Trade Secrets Act regarding proprietary company information would be an example of data that should not be made publicly available, as would detailed vendor pricing information in many cases. Note that when datasets are designated as inappropriate for public release due to, for instance, issues associated with personally identifiable information, agencies should consider whether the value of the data can be increased by either (1) making the data available to a set of qualified parties (e.g. researchers) with strong legal and privacy protections, or (2) providing access to other Executive branch agencies though through inter-agency data sharing agreements. However, M-13-13 still requires that these types of information be inventoried in an Enterprise Data Inventory regardless of whether they are considered “open data,” which may lead to valuable internal use. Your agency CIO as the responsible official for implementation of the policy will have more information about any agency or program specific considerations that should be applied to data in your agency.
####Q8: Some of the data I collect cannot be shared publicly. How is this data affected by the open data policy? A: The open data policy is about more than just making data publicly available; it is also about managing information as a strategic asset within the enterprise. Even data that cannot be shared publicly should be received and stored in platform-independent, machine-readable formats whenever possible. In many cases, there are significant benefits to be derived from sharing data across the government, even when the data will not be publicly posted (See, for instance, OMB Memorandum M-11-02: Sharing Data While Protecting Privacy). As an example, detailed contract price data that can be shared within the government can provide a valuable touch-point as other agencies plan their procurements, but in some cases that data cannot be publicly shared. Similarly, data collected in conjunction with administering public benefits can be valuable to statistical agencies.
####Q9: Who decides whether data should be posted publicly? A: Program office personnel and grant officers will largely be responsible for recommending whether a given data set should be made publicly available. As the data owners, such personnel and officers are in the best position to determine whether privacy, national security, or other concerns would preclude public posting. While agency general counsel,grant and contracting officers, and security and privacy personnel, among others, may all be involved in reviewing whether or not a given dataset can be made public, the agency CIO is the responsible official in each agency for leading the open data effort, providing leadership and guidance on implementation.
####Q10: Once the decision to post data publicly is made, who is responsible for doing so? A: The agency CIO has the primary responsibility for managing and creating an inventory of all agency datasets as part of an enterprise-wide data inventory. The agency CIO is also responsible for creating a public data listing, as a subset of the full inventory. The public data listing does not include the underlying databases, but is simply a list of all datasets (e.g. title, description, point of contact, etc.) in the agency that are public or could be made public. Public data listings, under M-13-13, are required to be posted at agency.gov/data pages.
A: No, FPDS-NG and USAspending already provide public data access to their contents in open formats. Some system changes may need to be considered for other Federal agency platforms that house public data, and enable public data to be accessed or downloaded.
###Questions Applicable to Contracts
####Q12: Do I need to initiate modifications on my existing contracts to address the open data policy? A: No. The open data policy is prospective. For new awards, Federal contracts must ensure that the Government treats data as a valuable national asset, and structures any data-related deliverables to collect such data in formats that can be shared, regardless of whether a determination has been made as to whether the data should be made available to the public. There is no requirement to modify existing procurement agreements.
####Q13: For new contracts, what changes must I make to my deliverables or terms and conditions to address the open data policy? A: That depends on what the Government intends to buy. If data are to be included as a deliverable, and the deliverable is written to require submission of those data in an appropriate format (See Section A or contact your agency CIO for more details), there are no additional requirements. The open data policy is not designed to change what the government is buying, but is focused instead on how that information is delivered, and ensuring it can be appropriately reused.
For example, if a deliverable consists of an assessment or end-state report and no underlying data are required, the open data policy will not force the agency to add a data deliverable or alter the other terms and conditions of the contract. If, however, a deliverable includes data, the deliverable should be structured to require the delivery or export of the data in a machine-readable format. The agency has discretion regarding whether deliverables must be provided in platform-independent formats, but if the deliverable is not provided in such a format, the agency will need to take the extra step of translating it into a platform-independent format. Either solution is acceptable to reach the objective of open data.
As agencies contemplate the purchase of systems that will process data, the requirements in Section 3 of M-13-13 must be considered. These requirements provide for a life-cycle view of effective and efficient information management by requiring that information is collected in a way that supports downstream processing, and that systems are built to support interoperability and information accessibility, including regular access or exporting of the data as a standard requirement of such systems. Addressing these considerations early in the acquisition process will protect against the costly retrofitting that is often involved in retrieving data from legacy platform-dependent systems.
A: A number of tools are available to transform data from proprietary formats to platform independent formats. Project Open Data has collected a number of such tools. Many of these tools are easy to operate and require very little investment of time or effort. As such, they can be used either by a contractor (if the contract specifies delivery in a platform independent format) or by Federal employees (if the contract specifies delivery in a proprietary format). Your agency CIO is responsible for leading the open data initiative in your agency as well as working with the interagency working group on this effort. Your agency may determine that having the agency translate the delivered data is most cost effective so we encourage you to work with your CIO on this effort.
####Q15: Are there any Federal Acquisition Regulation (FAR) provisions that might affect an agency’s authority to publish data provided as a contract deliverable? A: While the scope of work and other contractual language concerning the required deliverables will determine what data is delivered under a given contract, the rights that the government obtains to the data that is delivered are generally covered by one of a number of standard contract clauses. For civilian agencies, procedures for utilizing these clauses appear within FAR subpart 27.4 – Rights in Data and Copyrights (48 C.F.R. 27.400 et. seq.,). For the Department of Defense, contracting officers are instructed by the Defense Federal Acquisition Regulation Supplement (DFARS) to use the DFARS coverage in subparts 227.71 and 227.72 in lieu of the guidance in FAR subpart 27.4. Other agencies may have supplemental acquisitions regulations that apply. The procedures contained in both the FAR and the DFARS are designed to implement the policy principles expressed at FAR 27.402. Specifically: (a) To carry out their missions and programs, agencies acquire or obtain access to many kinds of data produced during or used in the performance of their contracts. Agencies require data to— (1) Obtain competition among suppliers; (2) Fulfill certain responsibilities for disseminating and publishing the results of their activities; (3) Ensure appropriate utilization of the results of research, development, and demonstration activities including the dissemination of technical information to foster subsequent technological developments; (4) Meet other programmatic and statutory requirements; and (5) Meet specialized acquisition needs and ensure logistics support. (b) Contractors may have proprietary interests in data. In order to prevent the compromise of these interests, agencies shall protect proprietary data from unauthorized use and disclosure. The protection of such data is also necessary to encourage qualified contractors to participate in and apply innovative concepts to Government programs. In light of these considerations, agencies shall balance the Government’s needs and the contractor’s legitimate proprietary interests.
While these clauses address the data rights obtained by an agency, decisions regarding whether data will be published will be made in accordance with Q9 above, involving, as appropriate, consultation among the agency CIO, general counsel, contracting personnel, security personnel and privacy personnel.
####Q16: Do I need to modify existing grants or financial assistance awards to address the open data policy? A: No. The open data policy is prospective and looks at grants made after the publication of the Executive Order. For these new awards, Federal financial assistance and grant agreements must ensure that the Government treats data as a valuable national asset, and structures any data-related deliverables to collect such data in formats that can be appropriately shared. There is no requirement to modify existing agreements.
####Q17: For new financial assistance awards, what changes must I make to terms and conditions to address the open data policy? A: You are not required to make any changes to the terms and conditions of new awards, but depending on the situation, you may want to consider future changes per below.
A1: If data are included or expected to be provided to the Federal government as an outcome of a Federal award and those data are already in the right format (See Section B or contact your agency CIO for more details), there are no additional requirements for the award.
A2: If data are expected to be provided to the Federal government but are not in the correct format, the agency must make a determination about whether to require the recipient to report the information in the new format (which may require a change to the terms and conditions), or whether to make the change to the new format themselves (which would not). Your agency CIO is responsible for leading the open data initiative in your agency as well as working with the interagency working group on this effort. Your agency may determine that having the agency translate the delivered data is most cost effective so we encourage you to work with your CIO on this effort. A3: If data are generated with Federal funding, but providing data to the Federal government would not otherwise be an expected outcome of the Federal award this policy does not include any new requirement to collect it.
####Q18: What are the implications of the policy for new investigator-initiated scientific research grants? A: Although most investigator-initiated research grants are expected to generate data, those that do not include a specific provision requiring that the resulting data be provided to the government are not subject to the requirements of this policy. However, to increase the value of the agency’s investment, some grants already require that data be provided to the agency. Furthermore, it is likely that some agencies will begin providing additional guidance with respect to managing digital scientific data created under investigator-initiated grants as part of their response to the Office of Science and Technology’s Memorandum on Increasing Public Access to the Results of Federally Funded Research. Agencies, in conjunction with their CIOs, may determine that some requirements of the M-13-13 policy are appropriate for inclusion in future funding announcements (e.g., machine readable, platform independent).
####Q19: How difficult is it for agency personnel to transform machine-readable data into a platform-independent format? Is requiring data in a platform-independent format likely to drive up costs for recipients? Who can provide guidance on making the decision regarding who should convert the data? A: A number of tools are available to transform data from proprietary formats to platform independent formats. Project Open Data has collected a number of such tools. Many of these tools are easy to operate and require very little investment of time or effort. As such, they can be used either by a recipient (if the award specifies delivery in a platform independent format) or by Federal employees (if the award either does not specify or specifies delivery in a proprietary format). Your agency CIO is responsible for leading the open data initiative in your agency as well as working with the interagency working group on this effort. Your agency may determine that having the agency translate the delivered data is most cost effective.
####Q20: Is there OMB guidance that might affect an agency’s authority to post data generated with Federal financial assistance? A: For Federal financial assistance to state, local, and tribal governments please see the guidance from A-102 . For Federal financial assistance to nonprofit organizations including institutions of higher education, please see the guidance in OMB Circular A-110.
While this guidance address the rights obtained by an agency, decisions regarding whether data will be published will be made in accordance with Q9 above, generally involving, as appropriate, consultation among the agency CIO, general counsel, contracting personnel, security personnel and privacy personnel.