0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *

52 + = 60

What is Data Cleansing? Why do I need it?
Sep
03

Data Cleansing: What It Is (and Why You Need It)

In the digital age, data is almost as good as currency.

This trend is largely powered by the advancement of AI and automation. Machine learning requires massive amounts of data. Not only that, but if you’re going to create more efficient processes, you need more material to work with.

Having a stockpile of accurate data helps on the analog side, too. This includes good, old fashioned snail mail sales letters.

Better data makes customer outreach more reliable and personalized, which empowers sales teams and marketing departments.

However, more efficient data usage also amplifies the effect of bad data.

Thousands of emails might be sent or hundreds of prospects called before errors are identified and corrected.

Entire batches of data analytics results can be ruined by bad data.

That’s why good data management policies are critical.

A vital part of your data management ecosystem is data cleansing.

What is data cleansing?

Data cleansing is the process of combing through data, and correcting errors or completing missing information.

No matter how thorough your data collection systems are, it’s impossible to prevent errors from getting into your databases.

People make mistakes on data entry forms. People intentionally omit or enter invalid information. There’s just no way to account for every possibility.

Moreover, there are massive benefits to having huge stores of prospect and client data. New data collection methods are being created all the time to help exploit these benefits.

Then, the expansion of digital resources and ways that customers can engage with companies means that error-free data collection is probably impossible. If it is possible, it won’t happen for a long time.

That means data cleansing is one of the most important mechanisms in data management.

Obviously, you should still be vigilant in catching errors during data collection, but regularly cleansing your data is really the only way to prevent errors from causing problems in many parts of your organization.

The best way to keep your data error free is to employ efficient data cleansing tools.

How to cleanse your data

Even with data cleansing tools, cleaning your data is still a process.

Since you need to regularly clean your data, creating an easily repeatable data cleansing process will make cleansing new data and refreshing old data far more efficient and saves money in the long run.

Identify key data fields

The most important pieces of information will vary from organization to organization. It depends on your customers, product, marketing strategy, and even your employees.

In any case, the first step in creating a solid data cleansing process is identifying what information is most valuable to your company.

This step also helps you create data validation guidelines, which improve your data collection process. With your key data identified, you can minimize the amount of unusable data entries that appear in your databases.

This step alone improves the quality of your data.

Analyze your data

With your key data in hand, you can go to your data stockpile and identify the gaps.

The analysis phase of cleaning your data is also an opportunity to organize it and remove any data fields that you don’t really need.

If you’re using Excel or Google Sheets, you can create scripts and workflows to streamline and automate a surprising amount of this process.

Here are a few key things to do as you’re analyzing and organizing your data:

Remove duplicate rows.

Duplicate rows can cause problems if you need to import your data. So cleaning these out can save you headache later on.

Remove spaces and nonprinting characters.

Extra spaces (unicode character set values 32 and 160) and weird characters (unicode character values 0 to 31, 127, 129, 141, 143, 144, and 157) can cause issues for sorting, filtering, and searching.

Getting rid of these makes your data management life much easier.

Merge and split columns.

If you’ve imported data, make sure that it’s divided up the way you need it to be. For example, you may want to split a single name column into a first and last name column.

This will help you identify missing information later on.

Having your data properly organized will also increase the internal efficiency of your company, since people will be able to more easily search and find what they need in your databases.

Append missing data

This is where the first two steps really start to pay off.

If you’ve got your data organized and you know which information is most important to you, you can pretty easily process your data in batches.

You have a couple options here.

  • You can use a batch data appending service. Simply upload a CSV, TXT, or Excel file. After that, the service will append the missing data that you need to your list.
  • You can use an API to create an integrated data append process so that your data processing is more internal to your organization.

 

The cool thing about using an API, is that you can also embed the data completion capabilities into the customer facing side of your website or app. With some creativity, this offers opportunities to streamline and automate your data validation at the point of collection.

Automate

Using the same data append services that you used to complete your data, you can automate continued maintenance and cleaning of your databases.

This is especially easy if you use an API to integrate data appending into your internal tools. Simply create a script that runs the data append function at regular intervals.

One mistake companies make is running through the first three steps every once in a while, like a spring cleaning for their data.

The trouble with this method is that these companies end up working with bad data for a month or two or more until they go through their databases again.

Creating automation that will continually refresh your data will keep you working with accurate information all the time.

Manage

Even with a good system for cleaning your data in place, you still need to consistently manage your databases to maintain data sets that your automated tools can work with, while remaining relevant to your needs as your business evolves.

Here’s what you need to do to keep your data in good shape:

Data backup

This one goes without saying. Backup your data, or you could find yourself having to recreate or buy entire silos of data all over again.

Input validation

Input validation is your first line of defense against bad data. As we mentioned before, perfect input validation probably isn’t possible.

But crafting your opt in processes, creating scripts that disallow incorrectly formatted data to be entered, and establishing procedures for manual data entry can help minimize the amount of data cleansing you’ll need in the long run.

Data validation

This is actually kind of a subset of input validation.

Data validation pairs with input validation to create a sort of two-step data entry process. Essentially, what you want is a mechanism that checks newly inputted data before it actually gets put into use.

A great way to do this is to build a script that simply removes incomplete or invalid data from a database and creates a separate container of data that will eventually become the file that gets run through the data cleanser.

Cleansing

Yep. Data cleansing is a key element of good data management. Refer to steps one through four in this article for more details.

Aggregating

Data aggregation can be achieved using built-in functions in your data management software like Excel or Google Sheets.

Aggregating your data essentially prepares it for use based on what elements of the data set are important for your company and objectives.

Another benefit of aggregating your data is that it provides one more layer of protection against bad data, since missing or invalid data can cause errors in the aggregation process.

So, aggregating your data can tell you if you’ve got any more tidying up to do before you send the data over to the teams who are going to use it.

Filtering

This part will be pretty easy for data sets that you’ve constructed with your own collection methods, since you probably only collect the data that you need.

However, if you’re using imported data that you acquired from a secondary source, there’s a good chance that there’s a bunch of stuff in there that’s not particularly useful to you.

In this case you’ll need to filter the data and toss anything that’s not useful. This makes it much easier for your teams to work with the data and get what they need from it.

If you frequently work with imported data, it’s a good idea to create a way to automate this process or at least have a workflow to streamline it.

Merging

This one is pretty straightforward:

If you can, you should merge multiples of the same type of data into a single container, so that it’s easier to search and process.

Appending

We covered data append when we talked about cleansing. Append any missing data to your lists to create complete data sets.

Deduping

Deduping is just a super technical sounding term for removing duplicates of any data you have. Unless you have a specific need for duplicate data, it’s best not to have more than one of each (with the exception of your backups).

Transforming

Data transformation is converting data from one structure or format into another. Typically, this is most useful if you use imported data. Sometimes you’ll get data in a format that your systems can’t work with. So you’ll need to transform it to make it workable for you.

Often, the program you use to work with your data, like Excel, won’t have good data transformation capabilities.

Fortunately, there’s a wide range of data transformation tools for getting this done.

Standardization makes this process simple. Avoid working with multiple data formats whenever possible. It just complicates things.

So that’s data cleansing (and data management).

While data cleansing alone isn’t enough to keep your data in shape, it will prevent headaches and can save your business a lot of money.

Feel free to check out our data cleansing services. Then leave a comment and let us know how you keep your data in order!

 

Sep
25

3 Government Databases You Must Know About to Stay Safe

3 Government Databases You Must Know About to Stay Safe

For over a decade, the United States has been on high alert to terrorist attacks, just as other parts of the world have too. There are so many world conflicts taking place that a country must take the proper precautions to protect its citizens.

In an effort to keep our country safe, the United States Government has put together several government databases listing known criminals, terrorists, products, and companies that may not do import and export business with the U.S. This also means that individuals or entities in the U.S. are forbidden to do business with any person or business that have been blocked, denied, and debarred and appear in these databases.

Where Do I Find the Government Databases?

There are websites that one can access easily to view the government databases. There are over 80 known lists to cross check whether a person or entity has been banned from doing business with the United States. When a U.S. company is conducting business with a foreign entity, it is the company’s responsibility to do their due diligence to ensure the safety of its country.

The U.S. Department of Homeland Security’s site will provide several links to lists providing information on blocked and denied entities, as well as a debarred persons lists.

The agencies the site refers to are the Bureau of Industry and Security which will provide a Denied Persons List and Entity List

The Office of Foreign Assets Control which provides a Special Designated Nationals and Blocked Person List

The Office of Defense Trade Controls, which gives a Debarred Parties List

The U.S. Government Printing Office, which will provide the Federal Register. 

  • Each of these agencies is responsible for keeping its lists up-to-date. The purpose of the lists is to provide exporters with companies, entities, and persons sanctioned by the U.S. Government and are forbidden to export goods from the U.S. It is the job of the exporter to be sure they are conducting transactions using the proper procedures and that exports are authorized properly.
  • The U.S. Department of State is another site that can provide useful information to ensure the safety of the country. The site provides links to the following lists: The Foreign Terrorist Organizations List (FTO), which provides a list of names of individuals or organizations that are or have been known to conduct business with terrorist groups or themselves are part of a terrorist group. Once these targets are listed, they will not be allowed to travel in the U.S. and any financial accounts with U.S. institutions are frozen.The State Sponsors of Terrorism is a list of countries that have been directly involved with support of terrorist groups. Currently there are four countries on this list, which are Cuba, Iran, Syria, and Sudan. Executive Order 13224, which was signed by President Bush in 2001, designates that the U.S. Government can impede terrorist funding. They have the power to stop financial support networks and block assets of foreigners or business that have given support to terrorists or pose a threat of doing so. The last link this site provides is to the Terrorism Designations Press Releases page, which lists the announcements from the Office of the Spokesperson as new terrorists or organizations are identified.
  • The U.S. Food and Drug Administration (FDA) provides a list of companies and individuals that are debarred under sections 306(a), (b)(1) and (b)(2) of the Federal Food, Drug, and Cosmetic Act, which is published in the Federal Register. Not only does this site provide the names of the offenders, but it gives the effective date they were added to the list, how long they are barred, and in most cases, a document about the case is provided from the Federal Register.

All of the sites referenced not only have links to the various lists, but also provide detailed information on how the lists are determined and used. Ultimately, it is the responsibility of individuals and businesses to use reasonable care and due diligence when conducting business affairs with foreign countries.

Mar
28

Using Online Databases to Find Doctors

Online Databases to Find Doctors
If you’ve recently moved to a new region, or you just want to switch doctors, you can start using online databases to find doctors. Internet databases make the process of finding the right doctor easier. Many such databases now exist, and many are free to use.

Reliable doctor databases should tell you mostly everything you need to know in order to make an informed decision, including a given doctor’s;

  • specialty
  • educational background, training and place of residency
  • years of experience
  • insurance coverage

 Where do I start my search for a new doctor?

You might start your search for a doctor by looking at a basic database which supplies the items listed above, such as the one maintained by the American Medical Association.

You could also try the WebMD Physician Directory. The information WebMD’s directory provides is updated twice every year, and includes directions to the offices of each physician it lists. If you’re on Medicare, you’ll find a list of Medicare-enrolled doctors through the government-run website called Physician Compare. On this site you can search for doctors according to a number of different criteria. You may search, for instance, for a doctor by gender, or for doctors who speak languages in addition to English.

Once you have put together a short list of doctors who are available to you, you’ll probably want to delve deeper and examine quality of care issues for each name on your list. Vitals.com is one website that can help you with this task. Its review system asks volunteer reviewers to rate doctors on intriguing criteria, such as: How accurate are a doctor’s diagnoses? How well does she follow up? How easy is it to make an appointment with him? How’s her bedside manner? The descriptions of doctor experiences which patients provide are often very detailed here.

Online Databases to Find Doctors

The Rate MDs website makes for compelling browsing as well. For one thing, it includes regional “top ten lists”: for example, the top ten best dentists – as reviewed by patients – near Wichita, Kansas. And its store of patient reviews is enormous, among the most extensive of any doctor database. Users can even add their own photos of doctors to the site.

It’s also important to search the Internet for malpractice databases – that is, databases that compile lists of malpractice complaints formally lodged against doctors. For instance, Health Grades is a public database informing site visitors of doctors’ histories of malpractice claims, including court rulings and settlement facts and figures.

And the federal government runs a similar database called the National Practitioner Data Bank. However, the NPDB, which comprehensively tracks malpractice actions taken against doctors, has been the subject of controversy.

In fact, the Health Resources and Services Administration (HRSA), kept this database shut down for a time, a move supported by the American Medical Association. And even when the NPDB went online again in late 2011, it did not provide the names of doctors with histories of malpractice lawsuits. Rather, it simply referred to those doctors by identification numbers. Many public health advocates and journalists, however, have fought for full disclosure on this website regarding those accused of malpractice. Whichever way this battle ultimately gets resolved, you should look at this database or similar databases to see what information you can glean.

A few words of warning are in order. Beware of medical databases which aggressively try to sell you medical products or insurance plans, and databases which doctors must pay to get listed on. And remember that patient reviews, like all reviews, are highly subjective. Different people bring to their doctors different expectations. Further, in most cases there’s no way to confirm that an online reviewer who claims to have been a patient of a particular doctor actually was a patient of that doctor.

In short, using databases to find doctors is a great start. Try reading at least several reviews, as together they should provide an overall impression of a certain doctor. On the other hand, don’t place too much trust in any single review.

Mar
23

Databases For Churches

Databases For Churches: There are certain things every place of worship needs: among them a faithful leader, a worshipful congregation and a computer database.

databases for churches

If that last item sounds strange to you, consider all the benefits computer databases can offer a church. With databases, church leaders can keep lists of the entire congregation, allowing them to instantly see who has recently made major contributions requiring  thank-you letters. They can also see which parishioners are sick and in need of home visits and extra prayers, and even which members haven’t been heard from in a while. They can use a database to keep track, too, of which religious instructors have taken classes in child protection. The list is endless.

Easy Contacting

When the leaders of a church set up a database, they have an easy way to contact every parishioner, every volunteer and every religious education student. With a database, emailing a particular group of people is easy to accomplish. If you have to cancel a Thursday night adult Bible class at the last minute, for instance, you can reach the entire group with a click of the computer mouse. You don’t have to worry about neglecting somebody, and you also don’t have to worry about bothering people who are not members of that class with an extraneous group email.

Scheduling

Databases are ideal for creating detailed weekly, monthly or even yearly church schedules, sharing those schedules with everyone who needs or wants to see them, and updating them as often as necessary. And if your church ever holds an event without much advance notice – maybe one week the Pope suddenly decides to drop by – you can send an urgent notice to everyone on your database.

Studying Demographic Patterns 

Church databases also make it easy to study the demographics of church attendance, which in many faiths has been on the wane in the last twenty or thirty years. Indeed, you can discover patterns at a glance. For instance, maybe you find that once many of your parish’s young people get married, they rarely show up at Sunday services afterwards. In that case, you could develop a program specifically aimed at retaining young married couples. For instance, maybe you would make it a point to sit down with every couple before they hold their wedding ceremony in your church, and tell them how much you’d miss them if they were to stop showing up on Sundays. The personal touch often works wonders – you might even say “miracles.”

What to Look For

So what specific qualities should you look for in a database for your church? Among the most important qualities are that:

  • It’s affordable.
  • It’s compatible with both PC’s and Macs.
  • It’s web-based. (That’s so you won’t have to worry about maintenance.)
  • You feel comfortable using it. Remember, a database is supposed to make your life easier, not create additional challenges. To that end, you might want to test a number of databases before arriving at a final selection.

Examples

There are many fine databases for places of worship, but you might want to look in particular at Church Community Builder, OneBody, FellowshipOne, Connection Power, ChurchInfo and Stafftool.

Final Note

As you might already be aware, many senior citizens in your parish may not be especially adept at using computers, and seniors tend to make up a significant percentage of regular church attendees. As a result, many of your older members might not receive your electronic communications. Perhaps you would consider addressing this issue by offering a weekly computer course geared towards seniors at your parish hall or recreational center – you might find a volunteer instructor among your flock.