Synopsis
Feeds are used to provision user, course, and registration data (as well as syllabus content) into your Concourse system. Feeds can import large amounts of data almost instantaneously, saving your System Admins hours of work. Manually processing feeds allows your institution more control over when and how frequently feeds are processed, but this operation does require an individual to include feed file generation and processing in their regular workload. If your institution has IT staff who are able to support automated feed processing, this approach allows you to quickly and accurately populate your Concourse system data with minimal human intervention. Ultimately, feed processing streamlines the Concourse experience for all users, from System Admins to Instructors.
Key Terms
- System Data Feeds: feeds used to create and update essential data in your Concourse system; includes User Feeds, Course Feeds, Section Feeds, Registration Feeds, and Course Tag Feeds.
- Syllabus Content (Item) Feeds: feeds used to create and update syllabus items; each syllabus item in your Concourse system has a designated Item Feed that can be used to populate the item.
- Syllabus Copy Feeds: feeds used to copy one or more items from source syllabi to destination syllabi; copy feeds copy the entire item selected, including its subitems.
- Data Removal Feeds: feeds used to drop user registrations and mark users and courses for deletion; users and courses are not deleted by these feeds–they are only identified as system components that are eligible for deletion through a manual process.
- Domain Feeds: feeds used to create domains in your Concourse system. Domain feeds can be used to update the names of domains; the external IDs of domains must be updated manually.
- Delete Locked Item Feeds: feeds used to delete syllabus items that have been locked or are otherwise not able to be removed from your system; each syllabus item in your Concourse system has a designated Delete Locked Item Feed that can be used to delete the item.
- Feed Log: a record of all feeds that have been processed in your system in the past 30 days; the food log lists: the type of feed that was processed, a link to the feed file, the method of processing (manual or automated), the user who processed the feed (if manual), the time the feed was received by Concourse, the length of processing time (duration), and a link to the output report.
- Output Report: a report of all actions made in your system by the feed that was processed; the feed report identifies whether a system component was created, updated, skipped, or if an error occurred as well as the related line of the feed file (e.g., “Updated: description from ENGL_2376_123 to ENGL_2376_456 (line 3)”).
- Subitem: a lower-level or child item that can be added to a syllabus in multiples once the parent item has been added (e.g., Textbook is a subitem of Materials); subitems cannot be populated or updated with item feeds, but subitems are copied when a user processes a copy feed of an item containing subitems.
Important to this Decision
Before making decisions about how your institution uses Concourse, you should consider how stakeholders, systems, timelines, and other factors are impacted. Review the important elements below to be better prepared to make changes to your Concourse configuration or implementation.
Prior to making this decision, you should consult your institution’s SIS and LMS admins, as well as IT staff involved with data management.
If your institution plans to provision system data via feeds, the decision of whether to use manual or automated feeds depends upon the consistency and availability of data in other systems, the ability to use job scheduler software, and the available personnel to support feed processing.
This decision is typically made during the implementation phase, but your approach to feed processing can be changed afterward if your institution’s needs change. Many institutions start out with manual feeds to become comfortable with the process and ensure accuracy during implementation; when it is confirmed that feeds are being created and processed as expected, institutions often transition to automated feeds.
Data used to generate feeds may come from a variety of sources, including the SIS, catalog tool, websites, or a collection of past syllabi. SQL queries can be used to collect data from some tools, while others (e.g., websites, past syllabi) require manual collection. The automation of feed processing requires the use of job scheduling software. Automated feeds are sent as POST over HTTPS while manual feeds are uploaded directly to Concourse by individual users.
If your institution processes feeds manually, an individual may generate feed files via a SQL query of the Student Information System. Less commonly, the individual may generate the feed file by hand, such as when item content appears on websites or in systems that are not as easy to query. If manual feed manipulation is part of your process, we recommend using Notepad++, Excel, or Brackets.io to format and review data because those programs identify each row’s line number, making it easier to diagnose feed errors. Once feed files are saved as .txt files, a System Admin should:
- Navigate to Admin > Tools.
- Select feed processing on the Tools page.
- On the subsequent page, select the type of feed to be processed from the Type dropdown menu.
- Select File and then locate the .txt feed file on your device.
- Select Process to submit the feed to Concourse.
- Once processing is complete, review the output report for errors.
If your institution automates feed processing, you should first generate the feed file(s) the same way that you would for manual processing–usually via a SQL query of the SIS. Concourse accepts a POST request to https://yourdomain.campusconcourse.com/process_feed_file containing three parameters:
- Type (case-sensitive)
- File
- HMAC: HMAC-SHA256 digest of feed file contents, using feed processing shared secret as the key (Shared secret is shown on the Admin > Tools > Feeds page of Concourse.).
Regardless of the approach, processing feeds saves a tremendous amount of time over manual data creation/entry and allows end users to access and engage with their syllabi more quickly. You may choose manual feed processing if you wish to have more granular oversight of the process; you may choose automated feed processing if your institution uses a large number of feeds or processes feed frequently in order to save a System Admin time and effort.
Configuration Options
Concourse provides different options for processing feeds to support different institutional contexts. We recommend that you review these options carefully before deciding which one best meets your needs.
- Option A: Automated Feeds
- Option B: Manual Feeds
These options are not mutually exclusive; if your institution opts to automate feeds, a System Admin can still process feeds manually if special exceptions arise. If you automate feeds and occasionally process feeds manually, it is important to keep feed sequencing in mind to ensure that manual and automated feeds do not create duplicate data or overwrite existing data. If the choices below don’t fit your needs, reach out to Client Services for assistance.
- Approach: Automated feeds rely on processes that generate data files from your SIS or other systems and send those files automatically to Concourse on a cadence that you set without any additional human intervention. We recommend that feed files be generated and processed manually at first to ensure accuracy of data, then automation is tested in the sandbox before it is deployed in the production environment. IT personnel managing this process should have appropriate permissions in the sandbox and production environments to facilitate implementation and later troubleshooting as needed.
- Better For: This option is used most often by institutions that:
- process a large number of feeds, and/or
- process feeds frequently, and
- have IT personnel available to support the implementation of automated feeds
- Advantages and Disadvantages of Using Automated Feeds: Automated feeds can be processed at any time of day or night, as many times in a day as is needed to support your institutional processes. Automated feeds are especially useful during the first days or weeks of a new term when courses and course information may be updated hourly; rather than a System Admin having to disrupt their workflow to manually process feeds repeatedly throughout the day, automated feeds can create and update courses and other system data like clockwork. One of the few downsides to automated feeds is that data from your SIS is sent to Concourse without human review, so if there is an error in the source of truth, that error will appear in Concourse. Even if the error is addressed by manual editing, if the automated feeds continue to be processed with the error, the error will overwrite manual edits and present in the interface.
- Effort: After the initial development process, using automated feeds requires little manual intervention. It is a good idea for high-level users to review feed generation and formatting practices as part of regular system maintenance to ensure that system data is being created as expected.
- Approach: Every step of this operation is guided by human intervention. A System Admin may generate a feed file by hand (not recommended due to incidence of error) or via a SQL query. Once the feed file is complete, the user uploads the file to Concourse and reviews the output report to confirm the data was created or updated correctly.
- Better For: This approach is better for institutions that:
- process a limited number of feeds,
- process feeds infrequently, or
- do not have IT personnel available to support the implementation of automated feeds.
- Advantages and Disadvantages of Using Manual Feeds: An advantage of manually processing feeds is that your institution has more oversight and awareness of the operation and is likely better prepared to troubleshoot and address errors as they arise. A disadvantage is that processing feeds manually does take an individual’s time, and the individual has to remember to process feeds at the right time of the term and in the right sequence. Like any other manual process, manually processing feeds is open to human error.
- Effort: The time investment of manually processing feeds varies based on the number and frequency of feeds being processed; actually processing a single feed takes only a matter of minutes on-demand. Another factor is that the method of generating feed files impacts the time and effort involved; feed files generated with a SQL query, naturally, take less time to develop and format than feed files generated manually.
Post-Decision Changes
Your institution’s needs may change over time and you may need to make changes to your configuration settings. In other words, the configuration decisions you make during the implementation phase might not be as efficient five or ten years afterward. Review the guidance below to better understand how changing this particular decision may affect your Concourse system or deployment in the future.
If your institution initially processes feeds manually and later decides to automate feed processing, you will need to:
- Communicate with the System Admin about the timeline for implementing automated feeds to ensure that manual feeds can be stopped appropriately to prevent duplicative or erroneous data creation.
- Communicate with your IT team to implement feed generation and processing via job scheduling software.
- Be prepared to communicate the following with the IT team:
- Concourse feed documentation
- The types of data needed for each feed
- The frequency, timing, and sequencing of feed processing.
- Ensure that your IT team has access to the Concourse sandbox.
- Test all new feeds in your sandbox prior to processing feeds in the production environment. Once you have confirmed that feeds are creating data as expected in the sandbox, you can start processing feeds in your production environment.
If your institution initially automates feed processing and later decides to process feeds manually, you will need to:
- Communicate with your IT team to develop a plan for discontinuing automated feeds. Your IT team may still be responsible for generating feed files, so it is important to be transparent about expectations and what is needed from IT moving forward.
- Designate at least one System Admin as the individual responsible for manually processing feeds.
- Clearly communicate to the responsible System Admin what feeds need to be processed, their frequency, and their sequence.
- Ensure the System Admin has been trained on processing feeds. If the System Admin hasn’t processed feeds before, a good place to start their training is with Concourse’s on-demand course all about Feeds.
- Confirm that the System Admin has access to the Concourse sandbox.
- Assist the System Admin as needed in the sandbox to ensure that they understand the operation and can process feeds correctly.
- Once you have confirmed that feeds are creating data as expected in the sandbox, the responsible System Admin can start processing feeds in your production environment.
Before making a decision about how to process feeds, you should be familiar with some related issues and contexts. See the following articles for more information:
- Concourse provides documentation about creating and processing different feed types. Review the articles that represent the types of feed you are interested in processing:
- Construct and Process System Data Feeds (user, course, section, registration, course tags)
- Construct and Process Domain Feeds
- Construct and Process Item Feeds
- Construct and Process Copy Feeds
- Construct and Process Data Removal Feeds
- Learn more about Automated Feed Processing.
- Skim through the Repository of Feed Errors to become familiar with troubleshooting feeds–a crucial part of learning to use data feeds.
Tell Us About Your Experience
Did this article answer your question? If you made a decision like this for your Concourse instance, let us know how it turned out. If we missed something in this article or if you have a question that isn’t addressed in our Knowledge Base, let us know how we can help by reaching out to support@campusconcourse.com.