Tagged with sub

Perform a Function on Each File in R

Sometimes you might have several data files and want to use R to perform the same function across all of them. Or maybe you have multiple files and want to systematically combine them into one file without having to open each file and manually copy the data out.

Fortunately, it's not complicated to use R to systematically iterate across files.

Finding or Choosing the Names of Data Files

There are multiple ways to find or choose the names of the files you want to analyze.

You can explicitly state the file names or you can get R to find any files with a particular extension.

Explicitly Stating File Names

fileNames <- c("sample1.csv", "sample2.csv")

Finding Files with a Specific Extension

In this case, we use Sys.glob from the base package to find all files including the wildcard "*.csv".

fileNames <- Sys.glob("*.csv")

Iterating Across All Files

We'll start with a loop and then we can add whatever functions we want to the inside of the loop:

for (fileName in fileNames) {

    # read data:
    sample <- read.csv(fileName,
        header = TRUE,
        sep = ",")

    # add more stuff here

}

For example, we could add one to every "Widget" value in each file and overwrite the old data with the new data:

for (fileName in fileNames) {

    # read old data:
    sample <- read.csv(fileName,
        header = TRUE,
        sep = ",")

    # add one to every widget value in every file:
    sample$Widgets <- sample$Widgets + 1

    # overwrite old data with new data:
    write.table(sample, 
        fileName,
        append = FALSE,
        quote = FALSE,
        sep = ",",
        row.names = FALSE,
        col.names = TRUE)

}

Or we could do the same thing, but create a new copy of each file:

extension <- "csv"

fileNames <- Sys.glob(paste("*.", extension, sep = ""))

fileNumbers <- seq(fileNames)

for (fileNumber in fileNumbers) {

    newFileName <-  paste("new-", 
        sub(paste("\\.", extension, sep = ""), "", fileNames[fileNumber]), 
        ".", extension, sep = "")

    # read old data:
    sample <- read.csv(fileNames[fileNumber],
        header = TRUE,
        sep = ",")

    # add one to every widget value in every file:
    sample$Widgets <- sample$Widgets + 1

    # write old data to new files:
    write.table(sample, 
        newFileName,
        append = FALSE,
        quote = FALSE,
        sep = ",",
        row.names = FALSE,
        col.names = TRUE)

}

In the above example, we used the paste and sub functions from the base package to automatically create new file names based on the old file names.

Or we could instead use each dataset to create an entirely new dataset, where each row is based on data from one file:

fileNames <- Sys.glob("*.csv")

for (fileName in fileNames) {

    # read original data:
    sample <- read.csv(fileName,
        header = TRUE,
        sep = ",")

    # create new data based on contents of original file:
    allWidgets <- data.frame(
        File = fileName,
        Widgets = sum(sample$Widgets))

    # write new data to separate file:
    write.table(allWidgets, 
        "Output/sample-allSamples.csv",
        append = TRUE,
        sep = ",",
        row.names = FALSE,
        col.names = FALSE)

}

In the above example, data.frame is used to create a new data row based on each data file. Then the append option of write.table is set to TRUE so that row can be added to the other rows created from other data files.

Those are just a few examples of how you can use R to perform the same function(s) on a large number of files without having to manually run each one. I'm sure you can think of more uses.

All the files are available on GitHub. You can see how eachFile.R, eachfile-newNames.R, and eachFile-append.R each do something different to the sample datasets.

An anonymous commenter pointed out that for big data files, he/she often uses: :::r setwd("my_working_directory_path") dat <- lapply(dir(pattern=".csv"), function(file) { dat.i <- read.csv(file) # ... data formating, subsetting, ... return(dat.i) } dat <- do.call("rbind", dat) I haven't tested it, but it is likely this is more efficient than using for loops in R.

Tagged , , , , , , ,

Truncate by Delimiter in R

Sometimes, you only need to analyze part of the data stored as a vector. In this example, there is a list of patents. Each patent has been assigned to one or more patent classes. Let's say that we want to analyze the dataset based on only the first patent class listed for each patent.

patents <- data.frame(
    patent = 1:30,
    class = c("405", "33/209", "549/514", "110", "540", "43", 
    "315/327", "540", "536/514", "523/522", "315", 
    "138/248/285", "24", "365", "73/116/137", "73/200", 
    "252/508", "96/261", "327/318", "426/424/512", 
    "75/423", "430", "416", "536/423/530", "381/181", "4", 
    "340/187", "423/75", "360/392/G9B", "524/106/423"))

We can use regular expressions to truncate each element of the vector just before the first "/".

grep,grepl,sub,gsub,regexpr,gregexpr, and regexec are all functions in the base package that allow you to use regular expressions within each element of a character vector. sub and gsub allow you to replace within each element of the vector. sub replaces the first match within each element, while gsub replaces all matches within each element. In this case, we want to remove everything from the first "/" on, and we want to replace it with nothing. Here's how we can use sub to do that:

patents$primaryClass <- sub("/.*", "", patents$class)

> table(patents$primaryClass)

110 138  24 252 315 327  33 340 360 365 381   4 405 416 423 426  43 430 523 524 
  1   1   1   1   2   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1 
536 540 549  73  75  96 
  2   2   1   2   1   1

This post is one part of my series on Text to Columns.

Citations and Further Reading

Tagged , , , , ,