Month: February 2024

For recruiters, the right recruitment tech gives them a competitive advantage. However, the methods used to gather online data on potential candidates can raise legal and ethical questions around data ownership and fair recruitment practices.

In this blog post, we explain the difference between capture and scraping, and how recruiters can best navigate this online frontier.

Capture vs. Scraping 

There’s no doubt that automating the process of extracting data about potential candidates or clients is a time-saver for recruiters. It’s a process that would otherwise be done manually, by transposing information from the source to a recruitment CRM or other data store.

From a recruiter’s perspective, what’s important is how that process is automated. This starts with knowing that capture and scraping are two very different approaches with very different implications. 

• A capture tool processes information displayed to an authorized user of a service after downloading to the memory of their device.   

• A scraping tool automates the download of prospective web pages directly to a database without user intervention.

Legally, this is yet to be codified, but ethically, the distinction is clear. A capture tool is designed to extract authorized information about a candidate from a source already accessible to the recruiter. Importantly, although it is automated, it still requires human direction and intervention. Essentially, capture is simply speeding up the manual copy/paste job that the recruiter would otherwise be doing.

On the other hand, scraping uses a ‘bot’ to indiscriminately extract candidate data from a website, which it may or may not be authorized to access. Authorization is largely determined by a data provider’s terms of service.

Terms Of Service

Put simply, some third-party terms of service distinguish between the use of legitimately downloaded information and the automated collection of information while others do not. 

At the same time, some terms of service are so restrictive that they essentially disallow any use of information not explicitly approved by the vendor. It’s important to note that very few terms of service agreements are ever tested legally all the way to trial and damages.

Third-party vendors also have all kinds of enforcement mechanisms outside the legal system, which can influence customer behavior. Whether used systematically or on a whim, it’s possible for vendors to take a variety of actions to deter actual or perceived malpractice, including warnings, altering service privileges, suspending users, or even cutting them off entirely. They may also choose to engage with tool vendors.

Legal Cases

hiQ LABS v. LinkedIn (now owned by Microsoft) is one of the most high-profile cases on scraping to pass through the US legal system. 

In 2022, the appeals court ruled that under the Computer Fraud and Abuse Act, anti-hacking law does not apply to public websites. So far, LinkedIn has been unable to obtain an injunction to stop hiQ from scraping personal data from its professional networking platform. 

So far, the merits have not been decided. Although it outlines an interpretation of the Computer Fraud and Abuse Act (CFAA), the appeals court has suggested that the CFAA is not the correct law to regulate hiQ’s conduct.

So where does that leave recruiters? Automated methods for extracting candidate information are here to stay. It’s only a question of time before a court ruling sets the precedent for the legitimate and ethical use of these tools. For now, it’s up to recruiters to use capture tools in accordance with relevant data privacy laws — such as the European Union’s General Data Protection Regulation (GDPR).

PCR Capture

As the name suggests, PCR Capture is a capture tool, not a scraping tool. As a specialist software vendor, its creator (Main Sequence Technology), has no measurable influence over major third-party service providers.

For that reason, we’re not able to certify any activity or obtain any kind of consent or waiver on behalf of our customers in their use of PCRecruiter Capture. Neither can we assume liability for behaviors and contexts over which we have no control.

However, we do assert a good faith belief that information downloaded to the memory of a user’s device in the normal and authorized course of the usage is information that belongs to the user, and the vendor has exhausted their right to that information by providing it.

Google’s new policies apply to emails received by any Google-hosted account, including Google Workspace and Gmail accounts. It is estimated that Google’s servers host roughly 1/3 of all email, so no matter what email provider you’re using to send your email, a significant number of your recipients are likely hosted by Google and will have their filters managed under these new rules. It’s important to understand these changes so you can stay on their good side.

Meet The Inbox’s bouncer

Spam filters exist to protect the recipient from unwanted and unsafe mail, and the rules about what is and is not allowed into the inbox are up to the email recipient’s provider and the email server administrator. This means there’s nothing that you can do to 100% guarantee your email won’t get routed to a recipient’s spam folder (if there was a surefire way to avoid every spam filter, the spammers would be using it.)

The email spam filter examines each incoming message for common hallmarks seen in previously reported spam, including text phrases, web addresses, sender domains, IP addresses, and underlying code contents. Each suspicious element found is used in a weighted calculation and mail that reaches a given threshold gets filtered out. The specifics of the formula are up to the email server, and some are beginning to apply GPT methods in addition to traditional tools. While Google Workspace Business edition admins can create whitelists and adjust spam settings for their own users, users of basic Gmail accounts get standard spam filter settings that are beyond their control.

These systems learn to identify spam based on which emails cause the user to hit the ‘spam’ button, as well as the emails that users choose to remove from their spam folders by hitting ‘not spam.’ They build up a reputation for your domain or IP that is continuously updated over time. Your objective as a sender should be to minimize the number of people who mark your messages as spam, as that can affect your deliverability overall.

6 Best Practices for Email Delivery

Whether you’re sending 10 emails or 10,000, there are established best practices that will go a long way toward keeping you deliverable. Several of them overlap with the CAN-SPAM Act, so adhering to them helps with your email getting delivered as well as your FTC legal compliance. The CAN-SPAM laws apply only to advertising email, so messages about an application process or other ‘transactional’ messages with existing candidates or clients are typically exempt from it. The laws in non-US jurisdictions may vary, so if you’re emailing people elsewhere, check the applicable laws.

1. Be thoughtful about what you send and to whom — Sending the same email blast to everyone in your recruitment database is bound to hit people it doesn’t apply to, and the less relevant people find your email, the more likely they are to flag it as spam. If you’re building an email sequence or sending a single-mail campaign, make sure your content and your email list are well-matched, and that you’re not including anyone unnecessarily.
2. Be transparent and authentic — Don’t fake your email address, use deceptive subject lines, or hide your contact information. Put your physical mailing address and phone number into the email. This not only legitimizes your business and helps to reassure the recipient that you’re not a scam, but it also gives them an alternate way to reach you if they want to do business.
3. Be responsive to opt-outs and keep your lists clean — Don’t make it difficult for people to stop getting your emails if they don’t want them. Include an unsubscribe link and make it easy to find. Remember that losing an unwilling email recipient is better than getting marked as spam, because getting a spam reputation affects your ability to reach willing recipients in the future. If someone opts out, retires, or bounces, make sure they stop getting your emails. It’s also smart to remove addresses from your lists if they haven’t clicked or replied to anything in a long time — outdated addresses can turn into spam traps. Buying email lists is also very risky and can fill your database with recipients who are more likely to bounce or to spam-flag you for cold emailing them.
4. Be clear and brief — A personal message from you to the recipient, particularly if you can mail-merge in their name or other unique info, is far likelier to make it past the inbox bouncer than a designed marketing piece with lots of ALL CAPS, common spam words, long subject lines, many colors and font sizes, or extraneous exclamation points. Avoid mixing advertising content and transactional content like offers or agreements in the same email. Believe it or not, some spam filters will even knock you for bad spelling or grammar, so proofread before sending! Before sending any mass email, send a test to yourself and see what it looks like in as many different email clients as you can manage. Unfortunately, there is no reliable HTML standard for formatted emails, and Outlook Classic for the desktop will often render margins and padding very differently than Gmail or Apple email, as well as blocking images. If your email doesn’t require fancy formatting and pictures, skip them for better consistency and deliverability.
5. Be prepared and informed — You can send your email to one or more free email spam checkers, such as M@ilgenius or Mailmeteor to see how they rate its deliverability. While these can catch mistakes or make suggestions before you send something, treat them as a guideline and not a guarantee — every mail server’s spam filter settings are different and a ‘good’ mail can still be blocked or a ‘bad’ mail might get through. You can also check your email domain or your IP address to make sure it’s not on any RBLs (remote block lists) using a tester like MX Toolbox. If you’re on a mail hosting service where the same email server is shared by multiple businesses, someone else’s spam habit could impact your delivery.
6. Be consistent — When an email server suddenly gets a lot of mail from a single source, particularly one that wasn’t already on their radar, it can appear suspicious and impact your deliverability. A mail server is less likely to filter out a regular newsletter than a sudden and singular blast of advertising. If you plan to send a large amount of email, ramp up slowly. Start with a small number of engaged recipients, and increase volume over time. The more email you’re planning to send, the slower you should take it as you’re ramping up. The same goes for email frequency — if you’re going to be sending emails daily, you’ll want to ramp up your quantities even more slowly than if you’re only going to send emails monthly.

What are Google’s new rules?

Google’s new changes are centered around sender authentication and stricter adherence to spam report thresholds. In response to rising scams and phishing emails, where someone impersonates a different sender by faking the “From” address on their email, Google is now changing “show our bouncer your ID at the door” from a recommendation to a requirement.

Their policies fall into two categories: rules that apply to everyone sending emails to Google accounts, and rules that apply specifically to bulk senders.

Google’s rules for everyone

Fortunately, most of the changes Google is making will not have much impact on you if you’re already using a properly configured email server and following the best practices above. We’ll outline the rules below and give some details on how they apply to PCRecruiter specifically.

Most of these items relate to technical settings that are handled by your DNS (domain registration) admin or your email provider. If you’re not sure who your configuration is through, you can perform a DNS lookup at MX Toolbox.

Here’s what Google requires for all senders:

• Keep your ‘spam rate’ below 0.10% and avoid ever reaching 0.30% or above. This is a ratio of emails from your domain that Google’s users have marked as spam vs. the number of emails received from your domain overall in the same period. You can check the stats for your domain by getting access to Google’s Postmaster Tools.

If you’re used to importing candidate lists or sending larger Campaigns in PCRecruiter, you’ll need to be even more particular about where you get your data from, what messages you send, and how often. Before these updates, Google treated the 0.30% spam rate as a recommended maximum, but these changes mean they could be treated with more scrutiny. If you want to get to the inbox, being more granular and relevant will be increasingly important. The Postmaster Tools also show your Domain Reputation — keeping your Spam Rate low will help to keep your reputation High.

• Authenticate your email domain with SPF or DKIM. These methods allow the recipient’s email system to verify that your email is really coming from the domain your ‘From’ address says it is from. Many public and private email servers already check these but Google will now be even more strict about them. 

You can verify your own SPF here. If you have a Gmail account, you can use the Show Original option in the More menu (three dots) on any email you’ve received to see if the sender’s SPF and DKIM passed or failed.

For PCRecruiter users who only send email directly from their own email account and server (i.e. you’ve linked PCR to your Google or Microsoft account), the authentication is already handled. However, if you’re using a dedicated bulk email service like SendGrid or using a self-managed email server, you’ll want to double-check that you’ve properly configured your DNS so that mail sent using your domain is showing up as authenticated, no matter what mail server it is sent from.

If you’ve got multiple Email Aliases in PCRecruiter, check SYSTEM > Email Setup > Email Alias List to review your outgoing settings. If the “SMTP Server:Port” box on any of them contains an IP address rather than a full email server domain, you may want to check with that provider about authentication options.

• Have valid PTR records for your domain. These forward DNS and reverse DNS records allow the recipient’s email server to verify that your mail is being sent from an IP address that is associated with your domain name. You can check your PTR here.
• Use TLS to secure your email communications. This is the modern standard for email connections. If you’re unsure whether you’re using TLS (you likely are), this can be checked by examining the email headers of a mail you’ve sent to yourself. You can sometimes tell by checking your outgoing mail settings in your email client as well.
• Make sure your emails follow Internet Message Format standards. These are the basic rules for how email content and code should be transmitted. You don’t have to do anything special for this rule, as it’s almost guaranteed that your email client is taking care of these requirements automatically.
• Don’t impersonate sending from a Gmail account. Sending email from a fake or mis-aligned email domain is bad practice, but faking Gmail’s own domain on a mail sent to Gmail would be a clear red flag.
• If you forward a lot of emails, check that your email system is using ARC, which makes sure that the SPF or DKIM authentication from the original email gets passed along when forwarding.

Google’s additional rules for bulk senders

Google has a few more requirements for “bulk senders” in addition to those listed above. Google defines bulk senders as any domain sending 5000+ emails per day to Google-hosted accounts. Keep in mind that this is cumulative, meaning that Google considers your domain a “bulk sender” whether you’re sending a single mass mail to 5000 recipients or 100 people at your company are sending 50 personal emails daily.

• Your emails must support a one-click unsubscribe header and include a clearly visible unsubscribe link in the message body. When an unsubscribe header is found in the email’s header code, Gmail will automatically add an “Unsubscribe” link to the top of the email near the sender’s info. They will also add an “Unsubscribe” button to the mail when mousing over it in the inbox list. Gmail will only add these two links for emails sent from addresses not found in the recipient’s Google Contacts.

PCRecruiter adds the required “one-click unsubscribe header” code to any outgoing Campaign or Bulk Mail that contains an unsubscribe link merge tag in the body automatically, so by using PCR’s opt-out features for your form letters you’re already compliant. When a recipient uses the ‘one-click’ unsubscribe, they’re instantly opted out of any future emails for that bulk mail category in your PCR database.

Hiding or omitting your unsubscribe link just makes it more likely that an unwilling recipient will mark you as spam, so making it easy to get off the list will improve your deliverability overall, whether you’re sending bulk emails to 5000 or 50. If you are not currently including an unsubscribe link in your bulk emails, review our documentation on setting up the opt-in/out features to get started. 

• In addition to having an SPF and DKIM as noted above, bulk senders must also set up a DMARC policy for the domain. DMARC gives you control over whether the recipient’s server should quarantine, reject, or do nothing special with emails from your domain if the SPF / DKIM checks fail. The main advantage to having this is that you’ll get reports about the emails that didn’t authenticate, which can help you identify problems or potential fraudsters sending mail that claims to be from you. Once you reach a point where you’re absolutely certain that everything you’re sending is always going to pass SPF / DKIM checks, then you may want to set the DMARC policy it to ‘reject’ so that any mail that doesn’t appear to be truly from you gets blocked. Make sure the “From” address on the email matches the domain listed in the DKIM or SPF record.

Be smart and fear not.

The typical PCRecruiter user is sending mail from a Google or Microsoft 365 account, which means a lot of the nuts and bolts of setting up authentication are already taken care of. PCRecruiter also already generates properly formatted emails and headers, so you don’t need to change anything to comply with these rules either. Plus with PCR’s bounce handling, opt-out list, and Sequencing, you can send the right emails to the right people and keep your database clean.

So, in essence, Google’s new rules just enforce what you should be doing already: sending legitimate emails to people who want to receive them and using a properly configured email server to do it. With a double-check of your domain record and a bit of extra scrutiny of your large-scale email plans, you should have nothing to worry about.