I’m trying to pull all the course names from a bachelor’s degree website. Here’s the code I’m using:
url = 'example-education-site.com/courses'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
course_titles = soup.findAll(class_='CourseTitle')
print(course_titles)
But when I run this, I just get an empty list. I’m not sure what I’m doing wrong. Are the class names correct? Maybe the site is using JavaScript to load content? Any ideas on how to fix this or a better approach to get the course names? I’m new to web scraping, so any help would be great!
hey mate, ive done some scraping before. sounds like ur dealing with dynamic content loaded by js. try using selenium instead of requests. it’ll render the page fully before scraping. also, double-check the class name - websites sometimes use weird naming conventions. good luck!
Hey Nate_45Guitar! Web scraping can be tricky, especially when you’re just starting out. Have you considered using a headless browser like Puppeteer? It’s pretty cool for handling JavaScript-heavy sites. 
I’m curious, though - have you tried inspecting the page source to see if the course titles are there? Sometimes websites use different class names for mobile vs desktop versions. Maybe that’s throwing things off?
Also, just wondering - is this for a personal project or school assignment? If it’s allowed, you could check if the site has an API. That might be an easier route to get the data you need.
Keep us posted on how it goes! What other cool projects are you working on with web scraping?