Legacy Code: Courses Search

Courses search is intended as an application that provides a more functional version of the official public facing course search app available at the address: https://my.sa.ucsb.edu/public/curriculum/coursesearch.aspx

Our version, currently deployed at https://courses.dokku-00.cs.ucsb.edu provides many more features:

For example:

• Search by instructor (What courses has Diba Mirza taught?)
• Search by course over a range of quarters (Who has taught CMPSC 130A over time?)
• and many more

UCSB_API_KEY values

To deploy proj-courses, in addition to the usual GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET needed for other OAuth apps in this course, you will also need a value for UCSB_API_KEY. This is a key that gives you access to the API for UCSB course information. These keys are obtained from the website https://developer.ucsb.edu.

You can request your own account, but it is typically faster to get one from the instructor, who will provide it to you on your team slack channel.

When setting up API keys, we typically enable all of the API endpoints that are not sensitive or protected (i.e. the auto-approved endpoints.) As of this writing, those are these endpoints:

• Academics - Events
• Academics - Academic Quarter Calendar
• Academics - Curriculums
• Dining - Dining Commons
• Dining - Dining Menu
• Students - Student Record Code Lookups

The ones that are currently used in the app are:

• Academics - Curriculums
• Students - Student Record Code Lookups

MongoDB Collections

The Courses Search application uses a MongoDB collection as backup storage for the UCSB Curriculum information, so you will also need to set up a MongoDB collection and put the URL for that collection in your environment variables. Read on for more explanation.

Why MongoDB for UCSB Courses Search

While it might seem redundant to cache curriculum information in a separate database (vs. just going to the API each time), doing so provides several important advantages:

• It may be faster to get data from the MongoDB collection than from the UCSB API for some queries
• Some query types are not supported directly by the UCSB API; in particular, the UCSB API does not support searches across multiple quarters; it can only retrieve data for one quarter at a time.
• It provides a backup in case the UCSB API ever becomes unavailable temporarily or permanently.

In addition, it provides an opportunity to learn about how to work with a MongoDB database.

Getting the value for MONGODB_URI

In the .env file for proj-courses, you’ll need a value for MONGODB_URI. That URI contains the hostname, username, password, and other information needed to connect to a MongoDB database provided by MongoDB.com.

To start, in order to keep things simple, we will use the same database credentials for localhost, dev, qa and prod. We acknowledge that this is not in keeping with “best practices” which would keep these separate (as they are for the SQL databases). You have the option of learning how to create your own databases on MongoDB.com and then getting separate credentials for these if this becomes an issue that you need to tackle.

To get the value for MONGODB_URI, here’s what you need to do.

1. Login to MongoDB.com; you should have gotten an invitation to a project in your email that matches your team name e.g. f23-7pm-1, f23-7pm-2, etc. under an organization that matches your class name (e.g. ucsb-cs156-f23).
2. Find your project. That should be a page that looks like this:
3. On the card for the deployment, there should be a Connect button, as shown below. Click that button
4. That brings up this modal. Click Drivers (the first option).

5. That takes you to a page like this one. The page asks you to choose a driver, but as it turns out that for both the default option (Node.js) and the actual option we are using (Java), the URI is exactly the same, and that’s all we need here. You will get the URI from the box as shown below. Note, however, that this is not the final form of the URI; we’ll have to do some editing here. But, go ahead and copy that URI and paste it into your .env file as the value of MONGODB_URI. For example, if the screen looks like this:

   Then in your .env file, you’ll start with this:

   MONGODB_URI=mongodb+srv://user:<password>@cluster0.vwrcszq.mongodb.net/?retryWrites=true&w=majority
   

6. Now, we need to do two things. The first one is to add in the name of our database, which we are calling database in order to keep things simple. The text database goes right after the single / and before the ?retryWrites, like this:

   • OLD: MONGODB_URI=mongodb+srv://user:<password>@cluster0.vwrcszq.mongodb.net/?retryWrites=true&w=majority
   • NEW: MONGODB_URI=mongodb+srv://user:<password>@cluster0.vwrcszq.mongodb.net/database?retryWrites=true&w=majority

   Note that by convention, our projects use database as the database name, but if you use something else (say potato) then this should go in this spot in the URI.

7. I said there were two things. The second thing is to replace <password> (including the <>) with the actual password. Note that while there is a password that’s already set for the username user, it is not possible to look that up; the only thing we can do is reset it. So the next few steps are to reset this password, and then copy that value in, replacing the <password> in the URI.

   Note that another option is to add addtional users with names other than user; each of those can have a different password. This may be important, because otherwise, if everyone on the team is sharing the same password, any time you need to update it, you have to coordinate with the entire team; changing it changes it for everyone. So, it’s up to the team how to manage this. Decide whether its easier to have just one user/password combination, or whether it’s easier to create separate user/passwords for each team member.

   To change the password value, navigate to the main page of the project, so that it looks like this:

   Find Database Access on the left nav and click. You’ll see a page like this one:

   

   Here, you have a choice; edit the user called user to reset the password, or create additional users. If you create additional users, you’ll need to change the user in the MONGODB_URI as well; for example, if you create cgaucho, then the MONGO_URI changes like this:

   • OLD: MONGODB_URI=mongodb+srv://user:<password>@cluster0.vwrcszq.mongodb.net/database?retryWrites=true&w=majority
   • NEW: MONGODB_URI=mongodb+srv://cgaucho:<password>@cluster0.vwrcszq.mongodb.net/database?retryWrites=true&w=majority

   In any case, to change the password, you go into the edit button beside the user. You’ll see this:

   

   Click the Edit Password button.

   Click Autogenerate Secure Password. Then either click the Copy button to copy the password, or click the Show button and copy it manually. Either way, it’s probably a good idea to Show the password and copy/paste it into your URI before you click Update User to make sure they match. And you must scroll down and click Update User! This is easy to forget because the Update User is not visible until you scroll down. I have made this mistaken dozens of times, and it’s a time-consuming one to make.

   Here’s what that looks like, including copying the password into the MONGODB_URI in .env and scrolling down to click Update User:

   

With that (along with configuring the other values in .env), you should be able to run mvn spring-boot:run

To check if it is working, login as an admin, and try loading courses for a quarter, like this:

As shown in the animation above, if the MongoDB database is working, you’ll be able to see messages indicating success.

At that point, you can also browse the MongoDB collection manually, as shown below.

Navigating to the databases and collections

Your team should have a deployment with the default name (typically cluster0). Inside this deployment, there should be a database called database, which can contain multiple collections.

Collections functions similar to tables in an SQL database, although they have a differerent structure: they are key/value stores that function similar to JSON objects. In fact, they use a variant of JSON known as BSON (which stands for “Binary Javascript Object Notation”). It’s the same idea, except, internally, the objects are not literally stored in JSON, but in binary encoding of JSON.

To navigate to your database and collections, follow this path in MongoDB.com:

From the deployment card, click Browse Collections:

You should see an interface similar to this one:

The collections themselves are explained in further detail below.

The main collection

The main database in your MongoDB deployment should be called database, and the main collection should be called courses.

You can do queries like as shown below. This is a query that finds all records for quarter 20231 (Winter 2023) and the enroll code 07443.

{ 'courseInfo.quarter' : '20231', 'section.enrollCode' : '07443' }

As we can see here, this brings up two records; this is the result of loading the data for this quarter twice without first defining an index to prevent duplicates (as illustrated below).

Avoiding Duplicate Data

To avoid duplicate data, it is helpful to define an indexes that prevents storing multiple documents with the same quarter and enroll code. Here’s how.

First, we want to ensure that the combination of courseInfo.quarter (e.g. “20231”) and section.enrollCode (e.g. 07443) is unique, so that we don’t end up storing duplicate data. We can do that by defining an index like this:

Go to the index tab (second over in MongoDB.com collections page):

Click the Create Index button at right:

That brings up this modal:

Fill in Fields with:

{
  "courseInfo.quarter": 1,
  "section.enrollCode" : 1
}

Then fill in options with:

{unique:true}

So that it looks like this:

Then click “Review”.

Note that if you already have duplicate data, adding this constraint will not eliminate those. You may need to drop the collection and recreate it with the constraint in place (i.e. add the constraint before you start adding data.)

Querying Data in MongoDB

The animation below shows how to do some basic queries in MongoDB.

A few tips:

• Use single quotes, not double quotes.
• Query by simply listing the keys (using dot notation for nested fields) and the values those fields should have.

• Examples:

  { 'courseInfo.quarter' : '20231'}
  

  { 'courseInfo.quarter' : '20231', 'courseInfo.title': 'INTRO DATA SCI 1' }
  

  { 'courseInfo.quarter' : '20231', 'courseInfo.title': 'INTRO DATA SCI 1' , 'section.enrollCode' : '07443'}
  

Queries with regex

You can use regular expressions to match course ids that start with a certain subject area. For example, this will find courses in quarter 20231 where the courseInfo.courseId field starts with MATH:

{ 'courseInfo.quarter' : '20231', 'courseInfo.courseId': { $regex: /MATH/ }}

Staff Setup for proj-courses

@channel Please look in your email for an invitation to create a MongoDB.com account, and join the project for your team (e.g. `f23-7pm-1`, `f23-7pm-2`, `f23-7pm-3`, `f23-7pm-4`).  Please accept that invitation before your next scheduled class, or during your next scheduled class. 

You will need this access in order to work on proj-courses. We'll explain more in class, and/or in future posts to this slack channel.

We'd like to see that all team members have done so by the end of the next scheduled class.