Construct and Process System Data Feeds

Construct and Process System Data Feeds

This article is designed for IT professionals or those who have a background in running feeds and just want the technical specifications for running system data feeds. If you would like more generalized information on the uses and benefits of Feeds, or you need access to tutorials on constructing and running Feeds, consider visiting the General Information articles and How-Tos for Feeds.

Overview:


This article will walk you through the specifications for the four System Data Feeds:
  1. User Feeds - add users to the Concourse platform.
  2. Course Feeds - creates a structured syllabus in the Concourse platform. 
  3. Section Feeds - add a section label to a syllabus.
  4. Registration Feeds - Register users to a syllabus.

These four feeds are considered the "primary" feeds in the Concourse platform because they are utilized the most, are at the heart of every implementation, and last but not least, all other feed types are typically secondary to the system data feeds. This article includes the Concourse feed specifications, samples for each of the feed files, and a debugging/error guide. They detail everything you should need to know to correctly construct and process the system data feeds in your instance of Concourse.

Though many of the definitions and parameters should be straightforward, below we walk through some important terms that are consistent for all feed types. You'll see these terms in any of the Feed specification articles, but it's always important to revisit them and work on memorizing them because they'll be a constant throughout every feed process.

Further, while there are many feeds that can be created and run, it is only necessary to construct the ones that are required to support the current scope of your implementation. Likewise, we suggest you begin experimenting with feeds in your sandbox that contain only a few rows to make sure that they are constructed properly before moving ahead to fully populated files in your production system.


Before You Begin: Important Feed Terms in Concourse

  • Source: You'll need to select the "source" for the data you wish to migrate. It is highly recommended that the data provided by these feeds be derived directly from their authoritative source (e.g. often the Student Information System). This is critical to easily integrate Concourse with other campus systems because you can potentially cut down on inaccurate or incomplete information.
  • Separators:  All feed files need to be a pipe "|" separated text file. Since much of this feed data should be obtained from a report or script within your SIS, IT should have no issue producing this for you. However, if you are to generate the data manually, note that this is different than a comma-separated value (CSV) worksheet that is typical in Excel. 
  • Updates:  Making changes to your feed file and processing them again will update all row data with some exceptions, such as the unique identifier(s) for a given feed row. For example,  if a new user, course, or section identifier is entered, it will create a new entry and not update the existing one. Feed-specific exceptions are noted in the feed specification.
  • Relationship:  Feed files require a degree of coordination and sequencing in order to properly work. You can explore feed coordination for system data feeds below in this article.

Construct the System Data Feeds

Below, you'll find the specifications for each of the System Data Feeds: User, Course, Section, and Registration. In each specification, we have provided:
  1. An overview of each feed and its purpose.
  2. The Row Header you can expect to use in the feed file.
  3. Definitions for the required row labels.
To actually construct a feed within Concourse, you will need:
  1. A plain text file program such as Notepad++
At the end of this article, you can download samples of each of these feeds. 

User Feed

User Feeds are run to add users to the Concourse platform.

Row Header:  USER_IDENTIFIER|FIRST_NAME|LAST_NAME|EMAIL|INTERNAL_USER

  • USER_IDENTIFIER:  This is the unique identifier for the user that is often used for common authentication (e.g. smithb). 
  • FIRST_NAME:  This is the first name of the individual being added. This first name should align with all other systems and should not be a nickname.
  • LAST_NAME: This is the last name of the individual being added. This name should align with all other systems.
  • EMAIL: This should be an insititution-specific email. It is highly recommended you do not use any external email addresses.
  • INTERNAL_USER:  This defines whether the user will be logging in using credentials stored in Concourse (=1) or those from a central campus system (=0), such as an LMS or portal.

Course Feed

Course Feeds creates a structured syllabus in the Concourse platform. There are a lot of fields that you can fill out on a Course feed, but not all of them are required. Below, we've bulleted the required information needed to successfully run a course feed, but you can use the optional fields as well. Taking advantage of optional fields can help with Advanced Search later.

Row Header:
Row headers are required for all Concourse feeds. If you attempt to run any feed without a header row, the feed will not process and you will receive a "Bad header row" error in the output report. In the Course Feed, however, some of the items in the header row do not have to be included in the data you add to the feed file. In other words, the feed file must include the full header row, but the course data rows can leave the fields blank. In the header below, the mandatory fields are highlighted. 

COURSE_IDENTIFIER | TITLE | CAMPUS_IDENTIFIER | DEPARTMENT_IDENTIFIER | START_DATE | END_DATE | CLONE_FROM_IDENTIFIER |TIMEZONE|PREFIX|NUMBER|INSTRUCTOR|SESSION|YEAR|CREDITS|DELIVERY_METHOD| IS_STRUCTURED | IS_TEMPLATE| HIDDEN_FROM_SEARCH

If you choose not to use the optional fields, they still need to be represented in the feed data rows. For example, a course feed that chooses not to use the TIMEZONE, INSTRUCTOR, SESSION, YEAR, CREDITS, and DELIVERY METHOD fields would appear as:

COURSE_IDENTIFIER | TITLE | CAMPUS_IDENTIFIER | DEPARTMENT_IDENTIFIER | START_DATE | END_DATE | CLONE_FROM_IDENTIFIER |TIMEZONE|PREFIX|NUMBER|INSTRUCTOR|SESSION|YEAR|CREDITS|DELIVERY_METHOD| IS_STRUCTURED | IS_TEMPLATE| HIDDEN_FROM_SEARCH

MUEB-124-SP24|Brass Ensemble|MAIN|MUEB|1/16/2024|5/3/2024|MUEB124-TMPL||MUEB|124||||||1|0|0​

  • COURSE_IDENTIFIER:  This is the unique identifier for the course and should be a sequence already used in the SIS (e.g. 2018_SP_ECON_1100_2A). Also, it is typical to use this to uniquely define the template set, which should be constructed in such a way that an SIS script can easily concatenate the parameters into a unique identifier (e.g. ACCT_4010_TMPL).
  • TITLE: This is typically the title of the course as seen in other external systems. (e.g. Advanced Microbiology).
  • CAMPUS_IDENTIFIER:  This needs to be set to one of the external identifiers provided on the Domain administration page. Please note that if we have relabeled your domain titles to match your organizational structure (e.g. "Location" instead of "Campus"), the feed still needs to use the original system (Campus) column title and data.
  • DEPARTMENT_IDENTIFIER:  This needs to be set to one of the external identifiers provided on the Domain administration page. Please note that if we have relabeled your domain titles to match your organizational structure (e.g. "Program" instead of "Department"), the feed still needs to use the original system (Department) column title and data.
  • START_DATE and END_DATE:  These are required for every course, even if they are not appropriate for something like a template. If that is the case, simply fill in a reasonable date range (e.g. the start of the academic year to the end). When templates are cloned, these dates are replaced with the then appropriate dates.
  • CLONE_FROM_IDENTIFIER:  Probably the trickiest of all the parameters, this will determine whether or not the course to be created should be from scratch or from another course using a course identifier already in the system. It is used most often when replicating a template for a given course section. This is why it is so important to create course identifiers for templates that can be scripted from the SIS (e.g. ACCT_4010_TMPL).
  • TIMEZONE: Choose the appropriate time zone for a given syllabus. This is especially helpful with satellite campuses that might be subject to another time zone. It is important to note that setting a time zone will impact how times display in a syllabus if the date and time template fields are used, as it will change on each syllabus based on the student's time zone. This field is optional.
Accepted Timezone Identifiers:
  1. Eastern clients should use: America/New_York
  2. Central clients should use: America/Chicago
  3. Mountain clients should use: America/Denver
  4. Pacific clients should use: America/Los_Angeles
  • PREFIX: This is typically the course code or acronym (e.g. Business might be BUSN), but this is typically standard across a given set of courses. This field is optional and is typically used for live courses.
  • NUMBER: This is the number associated with the course. This field is optional and is typically used for live courses.
  • INSTRUCTOR: The instructor teaching the course. This field is optional and is typically used for live courses.
  • SESSION: This is typically the term, or sub-term, for a given course (e.g. spring or fall). This field is optional and is typically used for live courses.
  • YEAR: This is the year the course is offered. This field is optional and is typically used for live courses.
  • CREDITS: This is the number of credits the course is worth. Typically, this is represented with a numerical value. It's optional and typically used for live courses.
  • DELIVERY_METHOD: The delivery mode is an optional field to designate if the class is traditional, online, hybrid, or any other delivery modality used at your institution.
  • IS_STRUCTURED:  This will determine if the created course will be based on the Concourse structure (=1), that is using the distinct fields for description, objectives, course policies, etc., or as simply a place to store uploaded syllabi (=0). Simplified, if you are using the items provided on a syllabus, it is a structured syllabus. File uploads are non-structured.
  • IS_TEMPLATE:  This field determines if a course is a template or not with a 0 or a 1. If the course should be a template, you'd designate it with a 1. If used in combination with the CLONE_FROM_IDENTIFIER and the course to be cloned is also a template, a linked template is then created, a very powerful but advanced concept. If the course should not be a template, or linked to a template (e.g. a live course), you would designate the field with a 0.
  • HIDDEN_FROM_SEARCH: In the Concourse platform, you can hide courses from search/advanced search by designating a course with a 1. This is not a typical operation and should be heavily considered because it will hide this course from all users in search, including administrators. You would need to run a report to find the hidden courses.

Section Feed

This feed is used to "insert" instances of sections within the courses created from the course feed. The reason this feed is separate from the course feed is to allow for multiple sections to participate in a single "master" syllabus. However, most institutions simplify their processes by creating one section per syllabus. Further, if you do not run this feed following the course feed, no sections will be available for users to be registered with.

Row Header:  COURSE_IDENTIFIER|SECTION_IDENTIFIER|SECTION_LABEL

  • COURSE_IDENTIFIER: This is the unique identifier for the course and should be a sequence already used in the SIS (e.g. 2018_SP_ECON_1100_2A). Also, it is typical to use this to uniquely define the template set, which should be constructed in such a way that an SIS script can easily concatenate the parameters into a unique identifier (e.g. ACCT_4010_TMPL).
  • SECTION_IDENTIFIER:  As noted above, many institutions will create one section per syllabus, which means this identifier and the COURSE_IDENTIFIER will look very similar. In this case, the course identifier may be 2018_SP_ECON_1100_2A (i.e. a course identifier really being used as a unique section identifier) whereas the section identifier is then 2018_SP_ECON_1100_2A_SEC (i.e. a dummy section identifier).
  • SECTION_LABEL: This is typically a numerical value that labels each section (e.g. 001, 002, etc.).

Registration Feed

This feed will register users, and their respective group, to a given syllabus. This feed is especially helpful for live-term courses or if you plan on registering a lot of individuals to many courses at once.

Row Header:  SECTION_IDENTIFIER|USER_IDENTIFIER|GROUP_NAME

  • SECTION_IDENTIFIER: The section identifier is what you created when you ran the section feeds. You cannot run a registration feed until you have run both section and user feeds.
  • USER_IDENTIFIER: The user identifier is the username for the individual you'd like to register to a given course.
  • GROUP_NAME:  This will determine into which group the user is placed when they are registered for a course. The seven available groups include: Managers, Developers, Assistants, Instructors, Students, Guests, and Public.

Relationship and Coordination of System Data Feeds

While you can technically run feeds in any order, there is a correct way to coordinate these feeds to ensure accuracy and the desired output:

  1. The  user feed  creates user accounts. It is normally the first feed to be processed. This feed can be run with any frequency.
  2. Second comes the  course feed , which creates new course syllabi.
  3. Next is the  section feed , which relates courses to course sections. It does  not  actually create individual sectional syllabi from course syllabi. Instead, it takes the courses already processed and applies one or more sections to them, all of which will share the same syllabus. You MUST run the section feed before registration feeds for live-term courses. However, you might not need to run section feeds during template creation if you do not plan on registering users to templates using feeds.
  4. Last comes the  registration feed , which ties together users with courses and their sections.

A Helpful Diagram

Coordination sequence for System Data Feeds

Let's take a quick look at this diagram to show the relationship and coordination between the system data feeds. When using this diagram, the sequence goes in "clockwise" order: User, Course, Section, Registration. First, we have the user feed because user feeds are typically the first feed run in any instance of Conourse, and they can be run in any frequency. They're simply adding users, and their details, to the platform. Next, we have course feeds. Course feeds create the syllabi en masse in the Concourse system. You can use course feeds for templates and live-term syllabi. You need to run this feed first because you can't add a section label to a syllabus that doesn't exist, nor can you register any users to the course. After you run a course feed, you can run a Section Feed * . In fact, when you run any course feed in the Concourse platform, it will remind you to run the section feed next. Section Feeds are important for live-term courses or any other courses that you would use registration feeds to assign users to a particular syllabus. Last, but certainly not least, we have the registration feeds. The registration feed registers a user, and their respective group, to a course section. So if you look at the black arrows, you can see the Course feed creates a course ID which carries down to the section feed. In the section feed, you have the section ID (which considers the course ID above) and then adds the labels you'd like. Following the black arrow over to the registration feed, you can see that in order to run the registration feed, you must know the section ID. Our User Feed is the only black arrow that goes counter-clockwise because the users must exist before you can register them to a given syllabus.

*Note: You do not have to run a section feed when creating templates unless you plan on using feeds to register users to your templates.

Now that we've gone through our diagram, a question that might come to mind is, "OK, so what happens if I accidentally break the proper sequence?" 
It's a fair question, and the exact outcomes can vary, but most likely, you'll receive errors when trying to run the feed or have incorrect/incomplete data processed. Ultimately, it's best to follow proper sequencing to save yourself time and potential troubleshooting headaches.

    • Related Articles

    • Construct and Process Data Removal Feeds

      This article is designed for IT professionals or those who have a background in running feeds and just want the technical specifications for running copy feeds. If you would like more generalized information on the uses and benefits of Feeds, or you ...

    • Construct and Process Item Feeds

      This article is designed for IT professionals or those who have a background in running feeds and just want the technical specifications for running system data feeds. If you would like more generalized information on the uses and benefits of Feeds, ...

    • Construct and Process Domain Feeds

      This article is designed for IT professionals or those who have a background in running feeds and just want the technical specifications for running copy feeds. If you would like more generalized information on the uses and benefits of Feeds, or you ...

    • Construct and Process Copy Feeds

      This article is designed for IT professionals or those who have a background in running feeds and just want the technical specifications for running copy feeds. If you would like more generalized information on the uses and benefits of Feeds, or you ...

    • Automated Feed Processing

      Concourse uses POST over HTTPS to allow for automated processing of feed files. This functionality allows you to control how frequently you run feeds and immediately receive information about what was successfully processed or had errors. To begin, ...