in Education by
I'm Running Ubuntu 16.04 LTS with Python 3.6.8 and I have the following code that allows me to iterate over lines in a file where I process each row and append the data to a database. I need to process a row, and then delete it or replace it with an \n or do anything to reduce the file-size of the text file. Also, I need at most 2 copies of the file: database and first-line-deleted file. with open(filename, buffering=1000) as f: for rows in f: #process text #delete row or replace with '\n' How exactly do I do this? JavaScript questions and answers, JavaScript questions pdf, JavaScript question bank, JavaScript questions and answers pdf, mcq on JavaScript pdf, JavaScript questions and solutions, JavaScript mcq Test , Interview JavaScript questions, JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)

1 Answer

0 votes
by
You have a big problem here: deleting the middle of a file isn't something you can do on most operating systems and their filesystems, and if you can, it's an esoteric operation with complicated restraints. So, the normal way to delete from the middle of a file, is to rewrite the entire file. But you seem to indicate in the comments that your file is hundreds of gigabytes. So reading the whole file, processing one line, and rewriting the whole file is going to be expensive and require extra temporary storage space. If you want to do this for every line, you'll end up doing far more work and require about double the amount of disk space anyway. If you absolutely have to do this, here are some possibilities: Read the file backwards and truncate it as you go. Reading it backwards is going to be awkward because not much is set up to help with that, but in principle this is possible and you can truncate the end of a file like this without needing to copy it. Use smaller files, and delete each file after you've processed it. This depends on you being able to change how the files are created, but if you can do it it's much simpler and lets you delete processed pieces sooner. On the other hand, do you definitely need to? Is the problem that the file is so big that the database will run out of room if it's still on the disk? Or do you just want to process more huge files simultaneously? If the latter, have you checked that processing multiple files simultaneously actually goes faster than doing the same files one after the other? And of course, could you buy more disks or a bigger disk?

Related questions

0 votes
    Actually I'm a newbie to the parsing stuff with Python Beautifulsoup4. I was scraping this website. I need ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 14, 2022 in Education by JackTerrance
0 votes
    I have made a chatroom where users can messages to each other but I want to add a upload file ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 20, 2022 in Education by JackTerrance
0 votes
    I am reading a yaml file like so in Python3: def get_file(): file_name = "file.yml" properties_stream ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Jun 1, 2022 in Education by JackTerrance
0 votes
    I have a dictionary d full of a collection of dataframes: key type size value gm1 dataframe mxn .. ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 26, 2022 in Education by JackTerrance
0 votes
    How do you convert single row from source into three rows into target?...
asked Mar 28, 2021 by JackTerrance
0 votes
    This question already has answers here: How are lambdas useful? [closed] (26 answers) Closed 3 years ago ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked May 15, 2022 in Education by JackTerrance
0 votes
    This question already has answers here: How are lambdas useful? [closed] (26 answers) Closed 3 years ago ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked May 14, 2022 in Education by JackTerrance
0 votes
    This question already has answers here: How are lambdas useful? [closed] (26 answers) Closed 3 years ago ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked May 13, 2022 in Education by JackTerrance
0 votes
    i want to create a new layer using scapy,i created a new layer but when i sent it to another ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 23, 2022 in Education by JackTerrance
0 votes
    I'm creating a button in Tkinter and I wanted to create rounded corners, other posts suggested using an ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 16, 2022 in Education by JackTerrance
0 votes
    I have a custom PyPi package. It is installed under Pyhon\Python38\Lib\site-packages\myCustomPackage. In the ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 7, 2022 in Education by JackTerrance
0 votes
    I have csv file and that looks like following. I want to remove all rows before one row values [ ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked May 14, 2022 in Education by JackTerrance
0 votes
    I want to log the user session. Currently the code is as follows (setting formatter and handlers is omitted ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Jul 11, 2022 in Education by JackTerrance
0 votes
    I want to log the user session. Currently the code is as follows (setting formatter and handlers is omitted ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Jun 30, 2022 in Education by JackTerrance
0 votes
    I want to log the user session. Currently the code is as follows (setting formatter and handlers is omitted ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Jun 23, 2022 in Education by JackTerrance
...