Attn computer nerds: What program to review a .txt file with 3M rows of data?
| Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | aoc | 09/25/24 | | baked sink burger | 09/25/24 | | effort | 09/26/24 | | maine is 180 | 09/25/24 | | CPI Inflation | 09/25/24 | | aoc | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | msj | 09/25/24 | | CPI Inflation | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | CPI Inflation | 09/25/24 | | .,.,..,,.,.,.,.,.,..,.,.,.,.. | 09/25/24 | | MartincourtMinicooper | 09/25/24 | | Taylor Swift is not a hobby she is a lifestyle | 09/26/24 | | .,.,..,,.,.,.,.,.,..,.,.,.,.. | 09/25/24 | | A lawyer (or lower) | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | msj | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | CPI Inflation | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | CPI Inflation | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | CPI Inflation | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | msj | 09/25/24 | | CPI Inflation | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | CPI Inflation | 09/25/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 | | peeface | 09/27/24 | | add | 09/25/24 | | kdh | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | rickpanama | 09/25/24 | | Kenneth Play | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | add | 09/25/24 | | rickpanama | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | rickpanama | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 | | Kenneth Play | 09/25/24 | | female penis | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/26/24 | | Kenneth Play | 09/26/24 | | msj | 09/26/24 | | Kenneth Play | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/26/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/27/24 | | msj | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/27/24 | | msj | 09/27/24 | | Kenneth Play | 09/27/24 | | Emotionally + Physically Abusive Ex-Husband | 09/27/24 | | Kenneth Play | 09/27/24 | | Emotionally + Physically Abusive Ex-Husband | 09/27/24 | | Kenneth Play | 09/27/24 | | David Poaster Wallace | 09/25/24 | | kdh | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | kdh | 09/25/24 | | my dog likes Haitian migrants less than I do | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | tim heidegger | 09/25/24 | | gtt | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | Ship Toaster | 09/26/24 | | coq | 09/26/24 | | jsm | 09/25/24 | | jsm | 09/25/24 | | CPI Inflation | 09/25/24 | | jsm | 09/25/24 | | aoc | 09/25/24 | | maine is 180 | 09/25/24 | | CPI Inflation | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | jsm | 09/25/24 | | wangfei | 09/25/24 | | Juicy J | 09/25/24 | | CPI Inflation | 09/25/24 | | Juicy J | 09/25/24 | | npb | 09/25/24 | | CPI Inflation | 09/25/24 | | yyz | 09/25/24 | | hex | 09/25/24 | | Carl von Clausewitz | 09/25/24 | | CPI Inflation | 09/25/24 | | Juicy J | 09/25/24 | | CPI Inflation | 09/25/24 | | Carl von Clausewitz | 09/25/24 | | posting in histrionic thread | 09/25/24 | | add | 09/25/24 | | LathamTouchedMe | 09/25/24 | | add | 09/25/24 | | David Poaster Wallace | 09/25/24 | | jsm | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | ceci n'est pas un avocat | 09/26/24 | | coq | 09/26/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 | | LathamTouchedMe | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | .,.,..,,.,.,.,.,.,..,.,.,.,.. | 09/25/24 | | David Poaster Wallace | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | CPI Inflation | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | fka | 09/27/24 | | Juicy J | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | .,.,..,,.,.,.,.,.,..,.,.,.,.. | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | .,.,..,,.,.,.,.,.,..,.,.,.,.. | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | Kenneth Play | 09/25/24 | | him | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | .,.,..,,.,.,.,.,.,..,.,.,.,.. | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | him | 09/25/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/26/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/26/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 | | him | 09/25/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 | | ChadGPT-5 | 09/25/24 | | .,.,..,,.,.,.,.,.,..,.,.,.,.. | 09/25/24 | | just be racist | 09/25/24 | | yyz | 09/25/24 | | wangfei | 09/25/24 | | just be racist | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | just be racist | 09/25/24 | | Diane Rehm talking dirty | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/25/24 | | Diane Rehm talking dirty | 09/25/24 | | Emotionally + Physically Abusive Ex-Husband | 09/26/24 | | rogue planet orbiting the SOL | 09/25/24 | | CPI Inflation | 09/25/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/26/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 | | https://imgur.com/a/IkQnGlr | 09/25/24 | | Diane Rehm talking dirty | 09/25/24 | | just be racist | 09/25/24 | | C'MON MAN! | 09/26/24 | | him | 09/25/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 | | MartincourtMinicooper | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/26/24 | | Baptist | 09/26/24 | | just be racist | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/26/24 | | him | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/26/24 | | David Poaster Wallace | 09/26/24 | | Juicy J | 09/27/24 | | Baptist | 09/27/24 | | Juicy J | 09/27/24 | | Baptist | 09/27/24 | | MartincourtMinicooper | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/26/24 | | just be racist | 09/26/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/26/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 | | Emotionally + Physically Abusive Ex-Husband | 09/26/24 | | him | 09/26/24 | | just be racist | 09/26/24 | | fka | 09/27/24 | | https://imgur.com/a/IkQnGlr | 09/26/24 |
Poast new message in this thread
Date: September 25th, 2024 3:58 PM Author: Emotionally + Physically Abusive Ex-Husband
Some assholes thought they'd be cute so just produced their entire sales database (including products we don't care about) as 6GB txt. file, with |-delimited fields.
Obv too big to open in Excel, but is there some other readily available software that can digest this and render it in a viewable/searchable form?
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131288) |
Date: September 25th, 2024 4:00 PM Author: aoc
this plugin for NotePad++ might work. I haven't tested it
https://github.com/superolmo/BigFiles
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131293) |
|
Date: September 25th, 2024 4:46 PM
Author: .,.,..,,.,.,.,.,.,..,.,.,.,.. ( )
Pandas is a python package. You can just import your data into a dataframe and then run whatever analysis you need on it. The syntax can be a bit obtuse, but it's one of the most extensively documented pieces of software on the planet. It's free and data scientists working on way more intense shit than you are use it every day. This is btw just one of many, many things pandas can do with data it's truly an incredible piece of software.
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131462) |
|
Date: September 25th, 2024 11:38 PM Author: MartincourtMinicooper
second sentence means that you'll never have 6M rows on the screen at once, so you never actually need to read that much
actually what does it mean to review millions of rows? you're not actually lookin g at them all, right? break the file up and open one of the smaller files, or write code that loads 100 rows into memory and only loads the next 200 if you hit pgdown or something
or just load the data into a database and then query that
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48132848) |
|
Date: September 25th, 2024 4:45 PM
Author: .,.,..,,.,.,.,.,.,..,.,.,.,.. ( )
/thread
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131460) |
|
Date: September 25th, 2024 5:32 PM Author: Emotionally + Physically Abusive Ex-Husband
"the concept of 'volume discount' seems like something that generally is legal."
This is the biggest hurdle we face with judges. I hand them a copy of the SCOTUS case with the relevant passage highlighted, and they're like, "Are you sure this is still good law??"
(A "volume discount" is legal so long as it is available to all competitors, and practically usable to them -- e.g., you can't have a tier with 50% lower pricing for 1M+ units)
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131657) |
|
Date: September 26th, 2024 2:45 PM Author: Emotionally + Physically Abusive Ex-Husband
"economies of scale may well be that 1m+ units or whatever justify a 50% price break"
In reality it never does, b/c the largest increment of shipment is the "truckload," so the cost savings for consumer goods actually plateaus pretty quickly.
But: The RPA anticipated exactly this concern with the "cost justification" defense -- if the supplier can show that its lower price to a given customer was in fact justified by its lower costs in supplying that customer, the discriminatory price is legal. In reality though, in every case we've had, the defendant's own data has shown that it's MORE costly to supply Walmart and Costco than to supply their independent competitors. The reason is that those guys have all sorts of packaging and specific delivery requirements, whereas for the indies the supplier just drops off a pallet at their loading dock at its convenience.
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48134822) |
|
Date: September 27th, 2024 7:18 PM Author: Kenneth Play
imo, most of the exceptions to your experience will be industries with high fixed costs and low marginal costs, right? eg think of any business that has to set up a factory, like paper and pulp production, semiconductors, steel, chemicals, smartphones, etc. once you get the factory set up and the product designed, the marginal unit cost is pretty low. but you could understandably want to give massive discounts to big purchasers who allow you to keep the lights on.
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48138852)
|
|
Date: September 27th, 2024 8:57 PM Author: Kenneth Play
"Where you have a large producer with low marginal costs, they don't realize a significant reduction in costs for the 100K-lot they send to Walmart vs the 100 1K-lots they sell to Walmart's small competitors."
If that's the logic of the RPA, I think it's retarded. In my example, no there's not strictly lower "cost" to the 100k lot, but the supplier needs them more because without them the entire factory becomes money-losing.
"A potential exception to my experience might be for sellers of expensive, customized products, like the heavy-truck seller in Volvo Trucks v. Reeder Simco."
Because the idea is, if you're selling a 100k lot you can set up custom assembly lines, etc to reduce marginal cost, where you simply can't do that for a one off sale?
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48139109) |
|
Date: September 27th, 2024 9:02 PM Author: Emotionally + Physically Abusive Ex-Husband
"but the supplier needs them more because without them the entire factory becomes money-losing"
First of all, fuck Walmart and fuck the manufacturer. Second, they can still sell to Walmart at whatever price they want, but they also have to give that price to Walmart's family-owned competitors. Seems like good public policy to me, and lowers consumer prices (the main thing we really care about) more broadly.
"if you're selling a 100k lot you can set up custom assembly lines, etc to reduce marginal cost, where you simply can't do that for a one off sale?"
Yeah, we haven't had a case of the latter type, but that's kinda my intuition from having taken Macro in 1997.
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48139123) |
Date: September 25th, 2024 4:13 PM Author: jsm
Can literally do it with commands.
1. sed Command
sed 's/|/,/g' input.txt > output.csv
2. awk Command
awk -F'|' -v OFS=',' '{ $1=$1; print }' input.txt > output.csv
3. tr Command
tr '|' ',' < input.txt > output.csv
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131349)
|
|
Date: September 25th, 2024 4:17 PM Author: jsm
You said:
If I upload a 6GB txt file, with |-delimited fields, can you convert it for me into a CSV file?
ChatGPT said:
ChatGPT
Yes, I can help convert your 6GB text file with |-delimited fields into a CSV file. Please go ahead and upload the file, and I'll process it for you.
You said:
Even if it has 3,000,000 rows?
ChatGPT said:
ChatGPT
Yes, I can handle large files with millions of rows, including those with 3,000,000 rows. The process may take some time depending on the exact file structure, but I'll be able to convert it for you. Feel free to upload the file, and I'll start working on the conversion to CSV.
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131362) |
|
Date: September 25th, 2024 5:45 PM
Author: .,.,..,,.,.,.,.,.,..,.,.,.,.. ( )
lol
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131714) |
|
Date: September 25th, 2024 9:40 PM Author: Emotionally + Physically Abusive Ex-Husband
A magistrate isn't going to say it's reasonably viewable because "Pandas is a python package. You can just import your data into a dataframe and then run whatever analysis you need on it. The syntax can be a bit obtuse, but it's one of the most extensively documented pieces of software on the planet. It's free and data scientists working on way more intense shit than you are use it every day. This is btw just one of many, many things pandas can do with data it's truly an incredible piece of software."
If it came to it, they'd say to the bald defensemos: "You're obviously able to export sales data, as shown by you having exported select data of Epah's clients when you filed that stupid motion to compel arbitration. So just export the parts he's asking for."
But as someone above said, easier just to work my way through this than spend 45 days to get to that ^^ result.
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48132451) |
|
Date: September 25th, 2024 5:45 PM
Author: .,.,..,,.,.,.,.,.,..,.,.,.,.. ( )
lol why would anybody need this? Python and pandas are such lightweight program that he should install his own version on his own computer. If somebody can't handle pip install pandas, the VM isn't going to do them any good.
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131710) |
|
Date: September 25th, 2024 5:53 PM
Author: .,.,..,,.,.,.,.,.,..,.,.,.,.. ( )
We're a bunch of autistic nerds on a dead internet forum. But seriously, if you need this much help, just get a code monkey from fiverr or whatever to handle this shit. You can do that right?
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131736) |
Date: September 25th, 2024 5:32 PM Author: him (🧐)
what do you need to review in the 3m rows of data? that's the real question. eg do you just need to ctrl-f 3 million rows for "bob's gay faggot co", or do you actually need to run queries on what's in the database, etc
nocode shit still has maybe another year or two before it's usable for your purposes (ie someone starting from 0 trying to do anything other than make a webapp)
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131656) |
|
Date: September 25th, 2024 5:58 PM
Author: .,.,..,,.,.,.,.,.,..,.,.,.,.. ( )
This is a trivially easy thing to do in pandas, see how cleanly it imports as a dataframe then filter by string contains.
I'm assuming this is some headerless CSV file? Your code would be as simple as something like
import pandas as pd
data = pd.read_csv('faggotdiscovery.txt', sep=",", header=None)
data.columns = ["Order Number", "Date", "Customer Name", "etc."]
filtered_df = df[df["Customer Name"].str.contains("JOHN SMITH")]
filtered_df.to_excel("JOHN SMITH.xlsx")
If you have a lot of names you could feed them to the script in a loop by reading out of excel or whatever, but that's more fancy than you need to get if we're only doing a a few people.
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131753)
|
|
Date: September 25th, 2024 6:11 PM Author: Emotionally + Physically Abusive Ex-Husband
Holy shit. Will you sign the PO attachment and do this for $46? I just need the sales to:
o WALMART WAREHOUSE
o COSTCO
o TWM
o VONS MARKET
o RITE AID
o CVS
o RALPHS
o SMART & FINAL
o ALBERTSONS
Plus our five guys.
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131797)
|
|
Date: September 25th, 2024 6:09 PM
Author: .,.,..,,.,.,.,.,.,..,.,.,.,.. ( )
(guy who has a hammer)
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48131788) |
Date: September 25th, 2024 10:08 PM Author: Diane Rehm talking dirty (🦆)
Many ways but this is a pretty easy one with no setup or config. You just need a little SQL know how.
Amazon Athena is an interactive query service that makes it easy to analyze data directly from Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to set up or manage and you can start analyzing your data immediately. You don’t even need to load your data into Athena, or have complex ETL processes. Athena works directly with data stored in S3.
https://aws.amazon.com/athena/
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48132557) |
Date: September 25th, 2024 11:02 PM Author: rogue planet orbiting the SOL
u could download Rstudio and R
and then import the txt file as a dataset
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48132725) |
|
Date: September 26th, 2024 5:26 PM
Author: https://imgur.com/a/IkQnGlr
You need to categorize each variable (e.g. nominal, ordinal, binary) before you can do anything with the dataset. It would help to know what kind of device/software generated it in the first place. When I worked with satellite pics it would have been no big deal to encounter a matrix with 3 million rows, since I was looking at output from digital sensors that were 3mp or greater. But I already knew what category of variable I was working with (categorical or interval), so it was easy enough to train a computer to automatically analyze the shit.
OTOH if I had to match each pixel in each row to some value in a relational database, no software I know of could do it out of the box, and you'd probably want to do it on the cloud or a $20k homebrew server.
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48135243) |
|
Date: September 26th, 2024 4:07 PM Author: MartincourtMinicooper
i asked this one thing to my AI just now:
> if i have a CSV file with 3 million rows, give me a python script to lazily read the file, only displaying the next line when I press a key
To achieve this in Python, you can utilize the csv module and input function for user input. Here's a script that will lazily read a CSV file, displaying the next line when you press a key:
python
import csv
def lazy_read_csv(filename):
with open(filename, 'r') as file:
csv_reader = csv.reader(file)
next(csv_reader) # Skip header if present
for row in csv_reader:
yield row
input("Press Enter to display the next line...")
# Replace 'your_file.csv' with the path to your CSV file
csv_file = 'your_file.csv'
for row in lazy_read_csv(csv_file):
print(row)
the whitespace is all fucked up thanks xo, but the original source looks correct to me; i only asked AI because i couldn't remember python standard library modules since i don't write in python often
this will display one line at a time, displaying the next only when you hit enter (i'd prompted any key, but it gave me enter; close enough)
you could change this to a pg-down like feature by having it read 20 lines at a time or something
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48135057) |
Date: September 26th, 2024 12:01 AM Author: Emotionally + Physically Abusive Ex-Husband
Response from Fiver bro:
“Okay sir
Let me see the link
Sorry for the late reply sir”
I could get used to this.
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48132963) |
|
Date: September 26th, 2024 2:50 PM Author: Emotionally + Physically Abusive Ex-Husband
"Okay Sir, you are my previous client and I will be happy to do this work but give me some moment i need to download the file as it is big it will take some time and also as those files are big so 100 is good enough I think"
"I will take some time to see the files and create the python script so please give me some time"
Maybe I should save him some time like, "my internet friends proposed the following script"
(http://www.autoadmit.com/thread.php?thread_id=5601362&forum_id=2\u0026mark_id=5310481",#48134838) |
|
|